AI Data Analyst

About the role

This role sits at the centre of response quality, competitive benchmarking, and release readiness. You will evaluate AI outputs, label and categorize failure modes, maintain high-quality benchmark data, and help ensure the datasets and evals we rely on are realistic, consistent, and tied to measurable product quality lift.

What you'll do

• Run ongoing human labeling for priority AI workflows, including response quality, task success, MCP and tool use, end-to-end Cowork-style workflows
• Triage bad-query reports, downvotes, escalations, and routed quality issues; categorize failure modes and feed clean analysis back to partner teams
• Perform qualitative analysis to identify recurring patterns such as hallucinations, retrieval failures, tool-use issues, weak grounding, and poor workflow completion
• Follow and improve labeling guidelines and rubrics so judgments are consistent, realistic, and useful for model and product improvement
• Maintain high-quality labeled datasets, golden sets, and regression slices; monitor coverage, drift, leakage, and difficulty distribution

What you need

• 3–5 yrs of experience
• SQL
• LLMs
• Data analysis
• NLP
• Machine learning
• Any graduate