Senior AI Evals Engineer

About the role

The Senior AI Evals Test Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK) or similar sort of agentic frameworks . This role bridges AI development and reliable deployment — ensuring our agents are safe, ethical, effective, and meet high-quality performance standards.

What you'll do

• Design and build agent evaluation frameworks — create end-to-end infrastructure for evaluating LLM-based agents across correctness, latency, cost, safety
• Build synthetic testing environments and simulation strategies to stress-test AI agents under diverse real-world conditions
• Establish eval-as-a-service — build and enhance centralized evaluation platform that any IT team members can self-serve to test their agents against standardized and…
• Automate regression and drift detection — implement CI/CD-integrated eval pipelines that catch regressions on every model or prompt change before production deployment
• Define eval taxonomies and rubrics — establish structured scoring criteria for agentic workflows (tool use accuracy, multi-step reasoning, hallucination rate

What you need

• Machine learning
• Python
• LLMs
• Prompt engineering
• MLOps
• GCP

Good to have

• SQL
• Pandas
• TypeScript
• Java
• AWS
• Deep learning