Azure Databricks Engineer- Visa Independent
• 5+ years’ experience in SQL (Expert)
• 4+ years of experience in Azure Databricks with PySpark
• 4+ years of experience in Azure Cloud platform
• 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL
• 2+ years of experience in Databricks workflow & Unity catalog
• 2+ years of experience in Python programming & package builds
• Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.
• Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).
• Ability to design modular, reusable workflows using tasks, triggers, and dependencies.
• Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
• Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.
• Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.
• Good to have Azure Entra/AD skills and GitHub Actions
• Good to have experience working on event-driven architectures using Kafka, Azure Event Hub
Responsibilities:
• Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics
• Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation
• Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms
• Implement and maintain ETL workflows for scalability cost effective and operational efficiency
• Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions
• Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.
Apply Now
Apply Now
• 4+ years of experience in Azure Databricks with PySpark
• 4+ years of experience in Azure Cloud platform
• 3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL
• 2+ years of experience in Databricks workflow & Unity catalog
• 2+ years of experience in Python programming & package builds
• Experience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.
• Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).
• Ability to design modular, reusable workflows using tasks, triggers, and dependencies.
• Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.
• Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.
• Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.
• Good to have Azure Entra/AD skills and GitHub Actions
• Good to have experience working on event-driven architectures using Kafka, Azure Event Hub
Responsibilities:
• Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analytics
• Write clean maintainable and efficient PySpark and Python code to support data ingestion transformation
• Integrate Azure Databricks with various Azure data services to build robust and scalable data platforms
• Implement and maintain ETL workflows for scalability cost effective and operational efficiency
• Collaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutions
• Participate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.
Apply Now
Apply Now