Lead AI QA Engineer

About the role

We’re looking for an Lead AI QA Engineer with 6+ years of experience who thrives on designing test strategies and evaluation harnesses for production-grade, agentic AI systems in addition to experience in ETL.

What you'll do

• Design and build evaluation harnesses for agentic systems in Python — golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions
• In addition, develop framework to verify generated AI output
• Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows
• Validate guardrails around tool execution: auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation
• Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships

What you need

• 6–11 yrs of experience
• Python
• SQL
• Performance testing
• LLMs
• Snowflake
• Unit testing
• Any graduate

Good to have

• CI/CD
• ETL
• Jenkins
• Git
• Docker
• MLOps