[Remote] Databricks Data Engineering Lead

Note: The job is a remote job and is open to candidates in USA. KANINI is seeking an accomplished Databricks Data Engineering Lead with deep expertise in the Azure Data ecosystem and healthcare data science. This role involves collaborating with data scientists and engineers to design, build, and optimize data solutions that enhance healthcare outcomes. Responsibilities • Lead the design and architecture of end-to-end data warehousing and data lake solutions, focusing on the Databricks platform, incorporating best practices for scalability, performance, security, and cost optimization • Design, build, and optimize scalable ETL/ELT pipelines in Azure for diverse healthcare data, including EHR, claims, IoT, and unstructured sources. • Implement advanced data storage and processing solutions leveraging Azure Data Lake, Synapse Analytics, Databricks, and Azure SQL. • Develop and optimize Delta Lake tables following the Medallion architecture (Bronze, Silver, Gold). • Orchestrate ETL workflows via Azure Data Factory (ADF) and Databricks Workflows. • Implement data access controls, account-level permissions, and object ownership via Unity Catalog. • Collaborate with architects, developers, and business stakeholders to ensure data quality, lineage, and governance. • Troubleshoot and optimize ETL pipelines for performance, scalability, and cost efficiency. • Support CI/CD integration using Databricks Repos, Git, and Azure DevOps. • Partner with data scientists to develop, deploy, and operationalize machine learning models and predictive analytics solutions supporting healthcare use cases. • Ensure data quality, governance, security, and compliance with HIPAA, PHI/PII, and other healthcare regulations. • Collaborate cross-functionally to translate business needs into robust technical solutions, aligning data engineering practices with healthcare standards (FHIR, HL7). • Contribute to the evolution of our data architecture strategy, incorporating best practices for cloud optimization, data governance, and emerging healthcare data trends. Skills • 8+ years of experience as a Data Engineer, with extensive work in Azure cloud services. • Strong proficiency in SQL, Python, PySpark, and Databricks for data engineering and advanced analytics. • Hands-on expertise with Azure Data Factory, Azure Synapse, Azure Data Lake, and Power BI. • Experience designing and implementing machine learning solutions in a healthcare environment. • Working knowledge of FHIR, HL7, and other healthcare interoperability standards. • Familiarity with DevOps practices and CI/CD pipelines for automating data workflows. • Strong analytical and problem-solving skills; proven ability to deliver in fast-paced, agile environments. • Experience with Big Data ecosystems (Kafka, Hadoop, etc.) for real-time and batch data processing. • Knowledge of data governance frameworks and compliance regulations in healthcare. • Microsoft certifications such as Azure Data Engineer Associate or Databricks certifications. • Understanding of cloud cost optimization strategies for large-scale data workloads. Company Overview • Redefining Excellence. Your Partner in Sustainable Software Development. It was founded in 2003, and is headquartered in Nashville, Tennessee, USA, with a workforce of 501-1000 employees. Its website is Apply tot his job