Senior Group Technical Architect

About the role

AI Evaluation & Observability Architect(CoE)- The AI Evaluation & Observability Architect makes AI systems measurable, reliable, and safe to run in production. The role designs the full evaluation lifecycle (offline and online evaluations, ground truth, golden datasets, deterministic and LLM-as-judge scoring) and turns it into enforceable engineering controls: regression gates, observability, SLOs, and resilience patterns.

What you'll do

• A. Evaluation Strategy & Design
• Define evaluation strategy per use case: what "good" means, which dimensions matter (accuracy, groundedness, relevance, safety, tone, latency, cost)
• Cover the full AI system stack: LLM calls, RAG/GraphRAGretrieva1, tool use, multi-step agents, and end-to-end workflows
• B. Ground Truth & Golden Datasets Build and govern ground truth and golden datasets: sourcing, labeling guidelines, subject-matter-expert review
• Generate and curate synthetic test data responsibly, with human validation and bias checks. Maintain dataset lifecycle: drift detection, refresh cadence

What you need

• 10–18 yrs of experience
• Git
• Python
• TypeScript
• AWS
• Microservices
• LLMs
• M.Sc or B.Tech or B.E. or B.Sc or BCA or M.Tech or MCA

Good to have

• MLOps
• Prompt engineering
• CI/CD
• Machine learning
• GCP
• Azure