[Remote] Senior or Staff MLOps Engineer – LLMOps

Note: The job is a remote job and is open to candidates in USA. TRM Labs is a blockchain intelligence company committed to fighting crime and creating a safer world. As a Senior or Staff MLOps Engineer with a focus in LLMOps, you'll be at the core of building and scaling the technical infrastructure for AI/ML systems, developing robust pipelines and operational tooling for AI applications. Responsibilities • Build reusable CI/CD workflows for model training, evaluation, and deployment — integrating Langfuse, GitHub Actions, and experiment tracking, etc • Automate model versioning, approval workflows, and compliance checks across environments • Build out a modular and scalable AI infrastructure stack — including vector databases, feature stores, model registries, and observability tooling • Partner with engineering and data science to embed AI models and agents into real-time applications and workflows • Continuously evaluate and integrate state-of-the-art AI tools (e.g. LangChain, LlamaIndex, vLLM, MLflow, BentoML, etc.) • Drive AI reliability and governance, enabling experimentation while ensuring compliance, security, and uptime • Build and enhance AI/ML Model Performance • Ensure data accuracy, consistency and reliability, leading to better model training and inferencing • Deploy infrastructure to support offline and online evaluation of LLMs and agents — including regression testing, cost monitoring, and human-in-the-loop workflows • Enable researchers to iterate quickly by providing sandboxes, dashboards, and reproducible environments Skills • Write high-quality, maintainable software — primarily in Python, but we value engineering ability over language familiarity • Have a strong background in scalable infrastructure, including: Containerization and orchestration (e.g. Docker, Kubernetes), Infrastructure-as-code and deployment (e.g. Terraform, CI/CD pipelines), Monitoring and logging frameworks (e.g. Datadog, Prometheus, OpenTelemetry) • Understand and implement ML Ops best practices, including: Model versioning and rollback strategies, Automated evaluation and drift detection, Scalable model and agent serving infrastructure (e.g. vLLM, Triton, BentoML) • Deploy and maintain LLM and agentic workflows in production, including: Monitoring cost, latency, and performance, Capturing traces for analysis and debugging, Optimizing prompt/response flows with real-time data access • Demonstrate strong ownership and pragmatism, balancing infrastructure elegance with iterative delivery and measurable impact Benefits • PTO • Holidays • Parental Leave Company Overview • TRM Labs is a software company that offers blockchain, transaction monitoring, and analytics to help financial institutions and governments. It was founded in 2018, and is headquartered in San Francisco, California, USA, with a workforce of 201-500 employees. Its website is Company H1B Sponsorship • TRM Labs has a track record of offering H1B sponsorships, with 1 in 2025, 4 in 2024, 3 in 2023, 3 in 2022, 1 in 2021. Please note that this does not guarantee sponsorship for this specific role. Apply tot his job