Senior Data Engineer (Python / Databricks)
ABOUT THE MISSION Relitix is currently seeking a hands-on Senior Data Engineer to lead a critical transformation of our data platform. We are moving away from legacy structures and re-engineering our ecosystem into a modern, Python-scripted Lakehouse architecture. This is not a maintenance role; it is a builder role for an engineer who values code-first engineering over GUI tools. If you are energized by owning systems end-to-end and want to build a platform that will scale significantly, we want to meet you. You will own the mission of migrating ADF-based pipelines to Databricks, ensuring internal BI data is accurate, and establishing secure governance via Unity Catalog. WHO YOU ARE We don’t just list responsibilities; we define clear outcomes. To achieve them, we look for specific traits in our engineers. These are the principles that will guide your success here: Code-First Mindset | Deep Ownership | Automation First | Governance Minded | Builder Energy The purpose of the Senior Data Engineer at Relitix is to modernize core architecture and secure the platform while ensuring reliable downstream BI. You don’t just spot problems; you fix them. Job Responsibilities • Modernize Core Architecture: Lead the full migration of priority pipelines into modular Python/Spark jobs running in Databricks, moving away from legacy ADF dependencies. • Optimize Performance: Re-architect pipelines to significantly reduce runtime and compute costs through partitioning, caching, and optimized orchestration. • Stabilize Data Ingestion: Architect and deploy robust Python connectors for API integrations (MLS, CRM, operational systems) with consistent logging and error handling. • Elevate Data Quality: Take ownership of internal BI datasets, implementing automated validation tests to ensure dashboards and operational metrics remain consistent and reliable. • Secure the Platform: Establish a governance foundation by implementing RBAC roles and Unity Catalog controls across bronze, silver, and gold data layers. Job Requirements • Strong Python Engineering: Strong Python engineering skills for ETL/ELT with a preference for writing clean, maintainable code over using drag-and-drop GUI tools. • Spark/Databricks Expertise: Experience with PySpark and SparkSQL inside Databricks, comfortable designing and building out Delta Lake and medallion architectures. • Governance Knowledge: Understanding of the importance of governance with exposure to Unity Catalog or similar frameworks for managing data permissions. • Automation Mindset: Approach problems with an automation-first mindset, utilizing Git-based workflows and CI/CD principles. • Accountability: High sense of ownership and accountability to ensure reliable downstream BI. Preferred Qualifications • Experience with data quality tools (e.g., Great Expectations). • Familiarity with MLflow or AI-ready pipeline structures. • Prior experience in multi-tenant SaaS data environments. • Madison, WI, Milwaukee, WI or Chicago IL area preferred Job Types: Full-time, Permanent Pay: $90,000.00 - $115,000.00 per year Benefits: • 401(k) • Dental insurance • Flexible schedule • Health insurance • Health savings account • Paid time off • Vision insurance Work Location: Remote Apply tot his job