Sr. Big Data Engineer
RedCloud Consulting is a global business and technology consulting firm supporting clients across North America, Europe, Asia-Pacific, and the Middle East. For more than two decades, RedCloud has partnered with high-growth mid-market companies and global enterprises to solve complex business and operational challenges—leveraging modern data platforms, artificial intelligence, and secure digital and physical infrastructure to drive smarter decisions and scalable outcomes.
Anchored in a foundation of integrity-based consulting, RedCloud brings senior-level experts together with client teams to deliver durable, high-impact solutions across Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. Recognized for both performance and culture, RedCloud has been named a top place to work and a fast-growing firm, and was recognized by Consulting Magazine as a 2024 Leader in Technology Consulting.
RedCloud seeks a Senior Big Data Engineer to support immediate client operations.
Overview:
The Senior Big Data Engineer will support core data platform delivery while driving engineering productivity through the development of internal accelerators and reusable tooling. This role is focused on real production impact rather than conceptual or research work. Ideal candidates are hands on engineers who thrive in high scale environments, care deeply about reliability, and enjoy building systems and tools that make other engineers more effective.
Responsibilities include but are not limited to:
Core Delivery Work
• Execute scoped data engineering workstreams across areas such as
• Technology migrations and consolidations across fragmented systems
• Security remediation for platforms and pipelines
• Data pipeline optimization to improve performance and reliability
Engineering Productivity Accelerators
• Design and build internal tools and automation that improve engineering velocity, including
• Tools for DDL creation and schema definition
• Airflow DAG scaffolding and workflow templates
• Spark configuration helpers and optimization utilities
• Standardized templates, generators, and reusable patterns to reduce repetitive work
Production Engineering
• Build and maintain high volume batch and streaming data pipelines
• Ensure reliability through monitoring, retries, and adherence to SLAs
• Operate effectively within enterprise environments with strong controls and security standards
Required Knowledge, Skills, and Abilities:
• Clear examples of building production grade ETL or ELT pipelines end to end
• Deep knowledge of Spark tuning including partitions, shuffle, joins, caching, executor sizing, and skew mitigation
• Strong understanding of streaming fundamentals including ordering, backpressure, idempotency, replay handling, and schema evolution
• Operational excellence in Airflow including operators, sensors, dependencies, SLAs, and backfills
• Strong DevOps hygiene including Git workflows, CI CD exposure, and code review participation
• Candidates should have experience creating tools used across engineering teams such as:
• Pipeline scaffolding generators or starter kits
• Standardized repository templates and recommended workflow patterns
• Automated DDL or schema generation tooling
• Configuration or Spark tuning automation frameworks
• Developer tools that eliminate repetitive tasks and increase throughput
• Candidates should be prepared to speak to what they automated, who used it, and what measurable impact it had.
• Core Technologies:
• Apache Spark including batch processing and performance tuning
• Apache Kafka including streaming and event driven pipeline patterns
• Apache Airflow including DAG authoring and orchestration best practices
• Programming Languages:
• Strong proficiency with Scala
• Experience with Java is also valuable
• Production and Systems Experience:
• High throughput data pipelines and distributed systems engineering
• Production grade reliability practices
• Strong understanding of data modeling, schema evolution, and workflow orchestration
• Cloud Environment:
• Ability to work confidently in cloud platforms outside of Azure such as AWS or GCP
Compensation range for position is $ 156,000 – 250,000 DOE.
Benefits and bonus information can be found at https://www.redcloudconsulting.com/careers
About Us:
RedCloud Consulting is a global business and technology consulting firm supporting clients across North America, Europe, Asia-Pacific, and the Middle East. For more than two decades, RedCloud has partnered with organizations ranging from high-growth mid-market companies to global enterprises, helping them solve complex business and operational challenges—leveraging modern data platforms, artificial intelligence, and secure digital foundations to drive smarter decisions and scalable outcomes.
Anchored in a foundation of integrity-based consulting, RedCloud brings together senior-level experts who work side by side with client teams to design and implement durable, high-impact solutions. Our approach emphasizes trust, accountability, and long-term value—balancing stability, growth, and innovation to help organizations operate securely and confidently in an increasingly complex global environment.
RedCloud delivers a comprehensive range of services across its core practices: Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. As our clients expand globally, so do we—providing consistent, expert support across regions while remaining agile, collaborative, and deeply invested in client success.
Recognized for both performance and culture, RedCloud has been named #1 Best Company to Work For in Washington (Mid-Sized Businesses) by Seattle Business Magazine, a “Fastest Growing Company” multiple times on the Inc. 500/5000 list, and recognized by Consulting Magazine as a Leader in Technology Consulting.
Visit http://www.redcloudconsulting.com/ for more info.
#LI-Remote
Apply tot his job
Apply To this Job
Anchored in a foundation of integrity-based consulting, RedCloud brings senior-level experts together with client teams to deliver durable, high-impact solutions across Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. Recognized for both performance and culture, RedCloud has been named a top place to work and a fast-growing firm, and was recognized by Consulting Magazine as a 2024 Leader in Technology Consulting.
RedCloud seeks a Senior Big Data Engineer to support immediate client operations.
Overview:
The Senior Big Data Engineer will support core data platform delivery while driving engineering productivity through the development of internal accelerators and reusable tooling. This role is focused on real production impact rather than conceptual or research work. Ideal candidates are hands on engineers who thrive in high scale environments, care deeply about reliability, and enjoy building systems and tools that make other engineers more effective.
Responsibilities include but are not limited to:
Core Delivery Work
• Execute scoped data engineering workstreams across areas such as
• Technology migrations and consolidations across fragmented systems
• Security remediation for platforms and pipelines
• Data pipeline optimization to improve performance and reliability
Engineering Productivity Accelerators
• Design and build internal tools and automation that improve engineering velocity, including
• Tools for DDL creation and schema definition
• Airflow DAG scaffolding and workflow templates
• Spark configuration helpers and optimization utilities
• Standardized templates, generators, and reusable patterns to reduce repetitive work
Production Engineering
• Build and maintain high volume batch and streaming data pipelines
• Ensure reliability through monitoring, retries, and adherence to SLAs
• Operate effectively within enterprise environments with strong controls and security standards
Required Knowledge, Skills, and Abilities:
• Clear examples of building production grade ETL or ELT pipelines end to end
• Deep knowledge of Spark tuning including partitions, shuffle, joins, caching, executor sizing, and skew mitigation
• Strong understanding of streaming fundamentals including ordering, backpressure, idempotency, replay handling, and schema evolution
• Operational excellence in Airflow including operators, sensors, dependencies, SLAs, and backfills
• Strong DevOps hygiene including Git workflows, CI CD exposure, and code review participation
• Candidates should have experience creating tools used across engineering teams such as:
• Pipeline scaffolding generators or starter kits
• Standardized repository templates and recommended workflow patterns
• Automated DDL or schema generation tooling
• Configuration or Spark tuning automation frameworks
• Developer tools that eliminate repetitive tasks and increase throughput
• Candidates should be prepared to speak to what they automated, who used it, and what measurable impact it had.
• Core Technologies:
• Apache Spark including batch processing and performance tuning
• Apache Kafka including streaming and event driven pipeline patterns
• Apache Airflow including DAG authoring and orchestration best practices
• Programming Languages:
• Strong proficiency with Scala
• Experience with Java is also valuable
• Production and Systems Experience:
• High throughput data pipelines and distributed systems engineering
• Production grade reliability practices
• Strong understanding of data modeling, schema evolution, and workflow orchestration
• Cloud Environment:
• Ability to work confidently in cloud platforms outside of Azure such as AWS or GCP
Compensation range for position is $ 156,000 – 250,000 DOE.
Benefits and bonus information can be found at https://www.redcloudconsulting.com/careers
About Us:
RedCloud Consulting is a global business and technology consulting firm supporting clients across North America, Europe, Asia-Pacific, and the Middle East. For more than two decades, RedCloud has partnered with organizations ranging from high-growth mid-market companies to global enterprises, helping them solve complex business and operational challenges—leveraging modern data platforms, artificial intelligence, and secure digital foundations to drive smarter decisions and scalable outcomes.
Anchored in a foundation of integrity-based consulting, RedCloud brings together senior-level experts who work side by side with client teams to design and implement durable, high-impact solutions. Our approach emphasizes trust, accountability, and long-term value—balancing stability, growth, and innovation to help organizations operate securely and confidently in an increasingly complex global environment.
RedCloud delivers a comprehensive range of services across its core practices: Data & AI, Delivery Excellence, Digital Solutions, Marketing & Sales, Privacy, Security, Data Center and Critical Infrastructure Security, and Strategy & Transformation. As our clients expand globally, so do we—providing consistent, expert support across regions while remaining agile, collaborative, and deeply invested in client success.
Recognized for both performance and culture, RedCloud has been named #1 Best Company to Work For in Washington (Mid-Sized Businesses) by Seattle Business Magazine, a “Fastest Growing Company” multiple times on the Inc. 500/5000 list, and recognized by Consulting Magazine as a Leader in Technology Consulting.
Visit http://www.redcloudconsulting.com/ for more info.
#LI-Remote
Apply tot his job
Apply To this Job