Data Engineer – AI
Job Description:
• Define and drive the technical vision for data platforms that support AI-powered features in Crossplane and Upbound Spaces
• Lead the design of data pipelines that transform infrastructure and data into training datasets for ML models
• Architect vector search and RAG systems that leverage Crossplane Control Planes & Upbound Marketplace as a knowledge store
• Build data infrastructure that processes resources, extensions, and compositions for semantic search
• Establish frameworks for collecting, processing, and analyzing infrastructure configuration data
• Design data pipelines that handle Crossplane-specific data
• Create infrastructure for indexing and searching Upbound Marketplace content, documentation, and community patterns
• Develop metrics and monitoring for AI features integrated with Upbound's control plane architecture
• Design data systems that power AI agents for infrastructure provisioning & operations, helping users generate and optimize Crossplane compositions
• Create feature engineering platforms that extract signals from control plane operations, resource status, and reconciliation patterns
• Implement data infrastructure for training models that predict infrastructure failures, optimize resource allocation, and suggest configuration improvements
• Drive the development of knowledge graph representations of infrastructure dependencies and relationships
Requirements:
• 10+ years of software/data engineering experience with at least 4 years in technical leadership roles
• Proven track record building data platforms that support production systems at scale
• Deep expertise in both traditional data engineering (Spark, Airflow, data lakes) and ML-specific infrastructure (feature stores, model serving)
• Experience with vector databases (Pinecone, Weaviate, Qdrant, Milvus, pgvector, Opensearch, ElasticSearch)
• Demonstrated experience with LLM applications, including RAG architectures and semantic search implementations
• Understanding of Kubernetes, cloud-native architectures, and infrastructure-as-code principles
• Strong understanding of data requirements for AI/ML systems: training pipelines, feature stores, and inference infrastructure
• Hands-on experience building knowledge bases and semantic search systems for technical documentation and code
• Experience with embedding models for code and technical documentation
• Knowledge of time-series data processing for infrastructure metrics and events
• Understanding of graph databases and their application to infrastructure dependency modeling
Benefits:
• Health insurance
• 401(k) matching
• Flexible work hours
• Paid time off
• Remote work options
Apply Now
Apply Now
• Define and drive the technical vision for data platforms that support AI-powered features in Crossplane and Upbound Spaces
• Lead the design of data pipelines that transform infrastructure and data into training datasets for ML models
• Architect vector search and RAG systems that leverage Crossplane Control Planes & Upbound Marketplace as a knowledge store
• Build data infrastructure that processes resources, extensions, and compositions for semantic search
• Establish frameworks for collecting, processing, and analyzing infrastructure configuration data
• Design data pipelines that handle Crossplane-specific data
• Create infrastructure for indexing and searching Upbound Marketplace content, documentation, and community patterns
• Develop metrics and monitoring for AI features integrated with Upbound's control plane architecture
• Design data systems that power AI agents for infrastructure provisioning & operations, helping users generate and optimize Crossplane compositions
• Create feature engineering platforms that extract signals from control plane operations, resource status, and reconciliation patterns
• Implement data infrastructure for training models that predict infrastructure failures, optimize resource allocation, and suggest configuration improvements
• Drive the development of knowledge graph representations of infrastructure dependencies and relationships
Requirements:
• 10+ years of software/data engineering experience with at least 4 years in technical leadership roles
• Proven track record building data platforms that support production systems at scale
• Deep expertise in both traditional data engineering (Spark, Airflow, data lakes) and ML-specific infrastructure (feature stores, model serving)
• Experience with vector databases (Pinecone, Weaviate, Qdrant, Milvus, pgvector, Opensearch, ElasticSearch)
• Demonstrated experience with LLM applications, including RAG architectures and semantic search implementations
• Understanding of Kubernetes, cloud-native architectures, and infrastructure-as-code principles
• Strong understanding of data requirements for AI/ML systems: training pipelines, feature stores, and inference infrastructure
• Hands-on experience building knowledge bases and semantic search systems for technical documentation and code
• Experience with embedding models for code and technical documentation
• Knowledge of time-series data processing for infrastructure metrics and events
• Understanding of graph databases and their application to infrastructure dependency modeling
Benefits:
• Health insurance
• 401(k) matching
• Flexible work hours
• Paid time off
• Remote work options
Apply Now
Apply Now