Data Scientist (AI Quality & Evaluation)
About the Role
We're looking for a Data Scientist to own the quality, reliability, and trustworthiness of our clinical AI outputs. You'll build the systems that ensure our AI "knows what it doesn't know" ā developing evaluation frameworks, calibrated confidence scoring, and automated quality assurance that physicians can actually trust.
What You'll Do
ā¢ Design and implement automated evaluation pipelines that assess AI output quality, accuracy, and safety at scale
ā¢ Develop uncertainty quantification systems where confidence scores meaningfully correlate with accuracy
ā¢ Build comprehensive evaluation frameworks combining automated assessment with clinician-validated test cases
ā¢ Implement feedback loops that continuously improve model outputs based on validation signals
ā¢ Establish scalable quality gates that catch errors before they reach end users
ā¢ Contribute to model alignment and fine-tuning efforts
Qualifications
Required
ā¢ Strong foundation in deep learning frameworks (PyTorch) and LLM architectures
ā¢ Experience with model evaluation, benchmarking, and quality metrics
ā¢ Proficiency in Python and modern ML development tools
ā¢ Strong statistical foundations
ā¢ Ability to read, implement, and extend research papers
ā¢ Excellent communication skills
Preferred
ā¢ Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)
ā¢ Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)
ā¢ Experience with RLHF, DPO, or preference optimization techniques
ā¢ Background in healthcare AI or regulated industries
ā¢ Experience building evaluation systems for production LLM applications
Apply tot his job
Apply To this Job
We're looking for a Data Scientist to own the quality, reliability, and trustworthiness of our clinical AI outputs. You'll build the systems that ensure our AI "knows what it doesn't know" ā developing evaluation frameworks, calibrated confidence scoring, and automated quality assurance that physicians can actually trust.
What You'll Do
ā¢ Design and implement automated evaluation pipelines that assess AI output quality, accuracy, and safety at scale
ā¢ Develop uncertainty quantification systems where confidence scores meaningfully correlate with accuracy
ā¢ Build comprehensive evaluation frameworks combining automated assessment with clinician-validated test cases
ā¢ Implement feedback loops that continuously improve model outputs based on validation signals
ā¢ Establish scalable quality gates that catch errors before they reach end users
ā¢ Contribute to model alignment and fine-tuning efforts
Qualifications
Required
ā¢ Strong foundation in deep learning frameworks (PyTorch) and LLM architectures
ā¢ Experience with model evaluation, benchmarking, and quality metrics
ā¢ Proficiency in Python and modern ML development tools
ā¢ Strong statistical foundations
ā¢ Ability to read, implement, and extend research papers
ā¢ Excellent communication skills
Preferred
ā¢ Master's degree in Computer Science, Machine Learning, Statistics, or related quantitative field (PhD preferred)
ā¢ Publications in top ML/AI venues (NeurIPS, ICML, ICLR, ACL)
ā¢ Experience with RLHF, DPO, or preference optimization techniques
ā¢ Background in healthcare AI or regulated industries
ā¢ Experience building evaluation systems for production LLM applications
Apply tot his job
Apply To this Job