Data Engineer (Digital Wallet) -US/ Hybrid
Role: Data Engineer (Digital Wallet)
Location: Scottsdale AZ (hybrid)
Duration: FTE & C2C
Job Description:
Must have : Skillset: Java, Scala, S3, Glue, aws , Redshift.
6-8 years of IT experience focusing on enterprise data architecture and management.
ā¢ Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
ā¢ Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
ā¢ Experience with Spark Scala and Java programming
ā¢ Data Lake concepts such as time travel and schema evolution and optimization
ā¢ Structured Streaming and Delta Live Tables with Databricks as a bonus
ā¢ Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation/support
ā¢ Advanced level understanding of streaming data pipelines and how they differ from batch systems
ā¢ Formalize concepts of how to handle late data, defining windows, and data freshness
ā¢ Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc
ā¢ Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermarks, etc.
ā¢ Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus
ā¢ Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness
ā¢ Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)
ā¢ Indexing and partitioning strategy experience
ā¢ Debug, troubleshoot, design, and implement solutions to complex technical issues
ā¢ Experience with large-scale, high-performance enterprise big data application deployment and solution
ā¢ Architecture experience in AWS environment a bonus
ā¢ Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus
ā¢ Experience with Gitlabs and CloudWatch and ability to write and maintain Gitlabs for supporting CI/CD pipelines
ā¢ Experience working with AWS Lambdas for configuration and optimization and experience with S3
ā¢ Familiarity with Schema Registry, and message formats such as Avro, ORC, etc.
ā¢ Ability to thrive in a team-based environment
ā¢ Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior-level of management
Apply tot his job
Apply To this Job
Location: Scottsdale AZ (hybrid)
Duration: FTE & C2C
Job Description:
Must have : Skillset: Java, Scala, S3, Glue, aws , Redshift.
6-8 years of IT experience focusing on enterprise data architecture and management.
ā¢ Experience in Conceptual/Logical/Physical Data Modeling & expertise in Relational and Dimensional Data Modeling
ā¢ Experience with Databricks & on Prem, Structured Streaming, Delta Lake concepts, and Delta Live Tables required
ā¢ Experience with Spark Scala and Java programming
ā¢ Data Lake concepts such as time travel and schema evolution and optimization
ā¢ Structured Streaming and Delta Live Tables with Databricks as a bonus
ā¢ Experience leading and architecting enterprise-wide initiatives specifically system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation/support
ā¢ Advanced level understanding of streaming data pipelines and how they differ from batch systems
ā¢ Formalize concepts of how to handle late data, defining windows, and data freshness
ā¢ Advanced understanding of ETL and ELT and ETL/ELT tools such as Data Migration Service etc
ā¢ Understanding of concepts and implementation strategies for different incremental data loads such as tumbling window, sliding window, high watermarks, etc.
ā¢ Familiarity and/or expertise with Great Expectations or other data quality/data validation frameworks a bonus
ā¢ Familiarity with concepts such as late data, defining windows, and how window definitions impact data freshness
ā¢ Advanced level SQL experience (Joins, Aggregation, Windowing functions, Common Table Expressions, RDBMS schema design performance optimization)
ā¢ Indexing and partitioning strategy experience
ā¢ Debug, troubleshoot, design, and implement solutions to complex technical issues
ā¢ Experience with large-scale, high-performance enterprise big data application deployment and solution
ā¢ Architecture experience in AWS environment a bonus
ā¢ Familiarity working with Lambda specifically with how to push and pull data, how to use AWS tools to view data for processing massive data at scale a bonus
ā¢ Experience with Gitlabs and CloudWatch and ability to write and maintain Gitlabs for supporting CI/CD pipelines
ā¢ Experience working with AWS Lambdas for configuration and optimization and experience with S3
ā¢ Familiarity with Schema Registry, and message formats such as Avro, ORC, etc.
ā¢ Ability to thrive in a team-based environment
ā¢ Experience briefing the benefits and constraints of technology solutions to technology partners, stakeholders, team members, and senior-level of management
Apply tot his job
Apply To this Job