Data Engineer — Common Data Environment (Databricks + AWS)

We are looking for a Data Engineer with strong experience in building scalable workflows and ingestion pipelines using Databricks and AWS. The role focuses on creating a centralized Common Data Environment (CDE) that integrates multiple data sources, automates workflows, and supports AI agents operating inside the environment. Clean, reliable engineering and documentation are critical. You will help implement: - Ingestion pipelines from multiple structured and unstructured sources into AWS S3 + Databricks - Delta Lake architecture (Bronze → Silver → Gold layers) - Unity Catalog setup and permissions management - Workflow orchestration using Delta Live Tables (DLT) or Databricks Workflows - Data quality checks, validation rules, and basic lineage - Transformations for general operational and analytical datasets - Integration with BI dashboards (QuickSight, Power BI, or similar) - Documentation for internal governance and environment readiness Responsibilities: - Build ingestion pipelines for various file types (CSV, Excel, APIs, JSON, etc.) - Implement and maintain Delta Lake tables and schema standards - Develop transformation notebooks and workflows in Databricks - Collaborate with the Data Architect on modeling and workflow design - Maintain version control (Git) and proper development practices - Add validation rules and logic checks in the data pipelines - Document pipeline logic, workflow dependencies, and data definitions - Join weekly project sync meetings - Recommend improvements for cost, scalability, and performance Required Skills - 3–5+ years of experience as a Data Engineer - Strong experience with: Databricks (SQL, PySpark, notebooks, workflows) Delta Lake + Unity Catalog - AWS S3, IAM, and cloud-native data workflows - Comfortable handling multiple data formats and sources - Strong documentation and Git workflow habits Nice to Have: - Experience building Common Data Environments (CDE) - Experience integrating BI dashboards - Exposure to AI/ML workflows or AI agent integration - Experience working in compliance-aware or structured environments Apply tot his job