Senior AI Product Ops

About the role

As AI agents move from POC to production, their behavior keeps changing in real time: the context they retrieve, the skills they select, how they resolve exceptions. These agents are never really “finished.” They keep learning from production outcomes, so their reliability, safety, and trustworthiness need to be actively operated, not just monitored.

What you'll do

• Own end to end production monitoring for one or more AI agents, tracking key metrics, KPIs, accuracy, latency, SLA breaches, guardrail triggers
• Serve as L1/L2 production support, providing first response monitoring, triage, and escalation to tech teams as new agentic services go live
• Identify and diagnose issues by monitoring eval scores and drift alerts, inspecting failing traces and patterns, and classifying root causes such as intent, tool
• Evaluate whether the agent’s reasoning remains correct in production, checking whether context retrieval stays current, whether resolutions match ground truth
• Monitor and manage post production reliability, including API and authentication failures and issues originating from external AI endpoints

What you need

• Machine learning
• LLMs
• Regression testing
• Any graduate