AWS Data Engineer – PostgreSQL to Databricks Migration
We are seeking a Senior Data Engineer with strong expertise in migrating PostgreSQL workloads to Databricks on AWS, specifically within the healthcare domain. The ideal candidate will be Databricks-certified, with a deep understanding of healthcare-specific data formats and compliance requirements. This role is critical to driving cloud modernization efforts for our healthcare clients, enabling secure, scalable, and high-performance data pipelines.Key Responsibilities:· Lead the migration of AWS-hosted PostgreSQL workloads to Databricks Lakehouse architecture.· Design and implement scalable ETL/ELT pipelines using Databricks (PySpark, SQL, Delta Lake) in AWS ecosystem· Work with complex healthcare data formats, including X12 837 claims, EBCDIC files, and other structured/unstructured formats.· Implement data masking, profiling, and governance solutions to ensure HIPAA-compliant data handling across ingestion, transformation, and consumption layers.· Optimize data pipelines for performance, reliability, and cost-efficiency within the AWS ecosystem.· Collaborate with data architects, analysts, and compliance teams to enforce data security best practices.· Drive data quality monitoring and reconciliation processes in high-volume, sensitive data environments.Must-Have Qualifications:· 8+ years of hands-on data engineering experience with strong focus on AWS + Databricks environments.· Proven track record of migrating PostgreSQL workloads from AWS to modern lakehouse platforms.· Databricks Certified Data Engineer (Associate or Professional).· Experience with healthcare data and regulatory formats including X12 837, HL7, EBCDIC files, etc.· Deep expertise in data masking, governance frameworks, and working in HIPAA-regulated environments.· Familiarity and working knowledge in adhering to CI/CD processes and practices in a cloud ecosystem.· Proficiency in PySpark, SQL, Delta Lake, and data transformation in Databricks.· Solid understanding of data ingestion from AWS (S3, Glue, Lambda) into Databricks.Preferred Skills:· Familiarity with FHIR, X12 837, HL7, or other clinical data standards.· Experience with Unity Catalog and Lakehouse governance frameworks.· Knowledge of Agile methodologies and CI/CD practices for data pipelines.· This role is healthcare domain-focused, and prior experience working with payers, providers, or healthcare clearinghouses is highly desirable. Apply tot his job