ML Model Evaluation Engineer

About the role

We are looking for an ML Evaluation Engineer to own model quality, regression testing, release validation, and production impact analysis for clinical AI systems. This role sits between applied ML, clinical data, MLOps, and production operations. Your job is to ensure that every model or workflow release is measurable, stable, and not degrading important clinical behavior.

What you'll do

• Build and maintain evaluation frameworks for clinical NLP, LLM, RAG, information extraction, and structured abstraction systems
• Create and manage hidden test datasets that are not directly visible to model developers, reducing overfitting risk
• Define release metrics, regression thresholds, slice-based evaluation, failure-mode tracking, and release/blocker criteria
• Compare model versions and identify performance degradation across clinical segments, document types, clients, data sources, labels, and edge cases
• Work with Clinical AI Data Specialists to design gold sets, hidden test sets, adjudication workflows, and label quality checks

What you need

• 3–6 years in data science & ai
• Machine learning
• NLP
• Python
• MLOps
• LLMs
• Regression testing
• Any graduate

Good to have

• Airflow
• Unit testing