Senior Technical Lead

About the role

We're looking for a Lead / Manager of AI Observability to own how we see, measure, and trust our AI systems once they're live in production. This role sits at the intersection of MLOps, platform engineering, and applied AI, and is responsible for building the monitoring, tracing, evaluation, and alerting infrastructure that gives engineering, product, and leadership real-time visibility into model and agent behavior, quality, cost, and risk.

What you'll do

• Own the end-to-end observability strategy for AI systems in production, including tracing, logging, metrics, evaluation pipelines, and alerting
• Design and build systems to monitor model/agent quality in production: accuracy drift, hallucination rate, latency, cost per request, token usage, and task success rate
• Establish golden signals and SLOs for AI systems, distinct from traditional infra SLOs (e.g., output quality, safety, groundedness, factuality)
• Build or integrate tracing across multi-step/agentic workflows so failures can be root-caused across prompts, tool calls, retrieval steps, and model versions
• Stand up automated evaluation frameworks (offline and online/production evals) to continuously score live traffic and catch regressions after model, prompt

What you need

• 3–6 yrs of experience
• Machine learning
• Python
• Go
• LLMs
• MLOps
• Incident response
• Any graduate