Big Data Platform Engineer/Data Platform Analyst (Azure, Spark, Airflow, Kubernetes)
Dice is the leading career destination for tech experts at every stage of their careers. Our client, ERPA, is seeking the following. Apply via Dice today!
Must Have Technologies:
• Azure
• Airflow
• Spark
• Hive
• Kubernetes
BDP Platform Engineer — Contractor Lead (Onsite)
Client: US Bank
Locations: Minneapolis, Atlanta, or San Francisco
Duration: 12-24 months
Overview: Lead and operate an Azure-based Big Data Platform using open-source tools (Airflow, Spark, Hive). Own onsite–offshore coordination, build reusable ETL frameworks, and ensure reliable dev/prod operations. Hands-on coding in Python, Java, and Scala with strong DevOps and IaC practices.
Key Responsibilities
• Platform engineering: Design, build, and run Azure data platform components with reliability, security, and cost efficiency.
• ETL frameworks: Develop reusable ETL/ELT libraries; standardize ingestion, transformation, testing, and deployment.
• Airflow & Spark: Author/maintain Airflow DAGs; implement and optimize Spark jobs; manage SLAs, retries, autoscaling.
• Onsite–offshore coordination: Establish handoffs, runbooks, and cadences; lead release orchestration and incident response.
• CI/CD & IaC: Build pipelines (GitHub Actions/Azure DevOps) and provision infra (HelmCharts) across environments.
• Observability & security: Implement logging/metrics/tracing; enforce IAM/RBAC, secrets (Key Vault), and compliance.
Requirements
• Experience: 4–8+ years in platform/data engineering with production ownership.
• Azure: ADLS Gen2, AKS, ACR, Event Hubs/Service Bus, Key Vault; Databricks or Synapse.
• Open source: Airflow, Apache Spark, Hive (metastore, table formats); strong SQL.
• Languages: Proficient in Python, Scala, and Java.
• DevOps/IaC: GitHub Actions/Azure DevOps; Terraform or Bicep; Docker/Kubernetes (AKS).
• Coordination: Proven leadership across time zones; vendor/stakeholder management; clear communication and documentation.
Tech Stack Highlights
• Azure (ADLS, AKS, ACR, Event Hubs, Key Vault, Monitor), Databricks/Synapse
• Apache Airflow, Apache Spark, Hive; optional Delta Lake/Iceberg
• GitLab Actions/Azure DevOps, Terraform/Ansible, Docker/Kubernetes
• Datadog
Thanks & Regards,
Anthony Kumar Guvvala
ERP Analysts, Inc.
Direct:
Email :
Apply Now
Apply Now
Must Have Technologies:
• Azure
• Airflow
• Spark
• Hive
• Kubernetes
BDP Platform Engineer — Contractor Lead (Onsite)
Client: US Bank
Locations: Minneapolis, Atlanta, or San Francisco
Duration: 12-24 months
Overview: Lead and operate an Azure-based Big Data Platform using open-source tools (Airflow, Spark, Hive). Own onsite–offshore coordination, build reusable ETL frameworks, and ensure reliable dev/prod operations. Hands-on coding in Python, Java, and Scala with strong DevOps and IaC practices.
Key Responsibilities
• Platform engineering: Design, build, and run Azure data platform components with reliability, security, and cost efficiency.
• ETL frameworks: Develop reusable ETL/ELT libraries; standardize ingestion, transformation, testing, and deployment.
• Airflow & Spark: Author/maintain Airflow DAGs; implement and optimize Spark jobs; manage SLAs, retries, autoscaling.
• Onsite–offshore coordination: Establish handoffs, runbooks, and cadences; lead release orchestration and incident response.
• CI/CD & IaC: Build pipelines (GitHub Actions/Azure DevOps) and provision infra (HelmCharts) across environments.
• Observability & security: Implement logging/metrics/tracing; enforce IAM/RBAC, secrets (Key Vault), and compliance.
Requirements
• Experience: 4–8+ years in platform/data engineering with production ownership.
• Azure: ADLS Gen2, AKS, ACR, Event Hubs/Service Bus, Key Vault; Databricks or Synapse.
• Open source: Airflow, Apache Spark, Hive (metastore, table formats); strong SQL.
• Languages: Proficient in Python, Scala, and Java.
• DevOps/IaC: GitHub Actions/Azure DevOps; Terraform or Bicep; Docker/Kubernetes (AKS).
• Coordination: Proven leadership across time zones; vendor/stakeholder management; clear communication and documentation.
Tech Stack Highlights
• Azure (ADLS, AKS, ACR, Event Hubs, Key Vault, Monitor), Databricks/Synapse
• Apache Airflow, Apache Spark, Hive; optional Delta Lake/Iceberg
• GitLab Actions/Azure DevOps, Terraform/Ansible, Docker/Kubernetes
• Datadog
Thanks & Regards,
Anthony Kumar Guvvala
ERP Analysts, Inc.
Direct:
Email :
Apply Now
Apply Now