Data Engineer - Databricks Specialist

We are seeking an experienced Data Engineer with deep expertise in Databricks to design, build, and maintain scalable data pipelines and analytics solutions. This role requires 5 years of hands-on experience in data engineering with a strong focus on the Databricks platform. Key Responsibilities: - Data Pipeline Development & Management - Design and implement robust, scalable ETL/ELT pipelines using Databricks and Apache Spark - Process large volumes of structured and unstructured data - Develop and maintain data workflows using Databricks workflows, Apache Airflow, or similar orchestration tools - Optimize data processing jobs for performance, cost efficiency, and reliability - Implement incremental data processing patterns and change data capture (CDC) mechanisms Databricks Platform Engineering: - Build and maintain Delta Lake tables and implement medallion architecture (bronze, silver, gold layers) - Develop streaming data pipelines using Structured Streaming and Delta Live Tables - Manage and optimize Databricks clusters for various workloads - Implement Unity Catalog for data governance, security, and metadata management - Configure and maintain Databricks workspace environments across development, staging, and production Data Architecture & Modeling: - Design and implement data models optimized for analytical workloads - Create and maintain data warehouses and data lakes on cloud platforms (Azure, AWS, or GCP) - Implement data partitioning, indexing, and caching strategies for optimal query performance - Collaborate with data architects to establish best practices for data storage and retrieval patterns Performance Optimization & Monitoring: - Monitor and troubleshoot data pipeline performance issues - Optimize Spark jobs through proper partitioning, caching, and broadcast strategies - Implement data quality checks and automated testing frameworks - Manage cost optimization through efficient resource utilization and cluster management - Establish monitoring and alerting systems for data pipeline health and performance Collaboration & Best Practices: - Work closely with data scientists, analysts, and business stakeholders to understand data requirements - Implement version control using Git and follow CI/CD best practices for code deployment - Document data pipelines, data flows, and technical specifications - Mentor junior engineers on Databricks and data engineering best practices - Participate in code reviews and contribute to establishing team standards Required Qualifications Experience & Skills: - 5+ years of experience in data engineering with hands-on Databricks experience - Strong proficiency in Python and/or Scala for Spark application development - Expert-level knowledge of Apache Spark, including Spark SQL, DataFrames, and RDDs - Deep understanding of Delta Lake and Lakehouse architecture concepts - Experience with SQL and database optimization techniques - Solid understanding of distributed computing concepts and data processing frameworks - Proficiency with cloud platforms (Azure, AWS, or GCP) and their data services - Experience with data orchestration tools (Databricks Workflows, Apache Airflow, Azure Data Factory) - Knowledge of data modeling concepts for both OLTP and OLAP systems - Familiarity with data governance principles and tools like Unity Catalog - Understanding of streaming data processing and real-time analytics - Experience with version control systems (Git) and CI/CD pipelines Preferred Qualifications - Databricks Certified Data Engineer certification (Associate or Professional) - Experience with machine learning pipelines and MLOps on Databricks - Knowledge of data visualization tools (Power BI, Tableau, Looker) - Experience with infrastructure as code (Terraform, CloudFormation) - Familiarity with containerization technologies (Docker, Kubernetes)