Lead AI QA Engineer
BengaluruFull-time6–11 yrsPosted 5 months ago
Senior QA and testing role at Kobie Marketing in Bengaluru. Needs 6+ years.
About the role
We’re looking for an Lead AI QA Engineer with 6+ years of experience who thrives on designing test strategies and evaluation harnesses for production-grade, agentic AI systems in addition to experience in ETL.
What you'll do
- Design and build evaluation harnesses for agentic systems in Python — golden datasets, LLM-as-judge graders, multi-turn regression suites and trace-based assertions
- In addition, develop framework to verify generated AI output
- Author automated test suites for prompts, tools, structured outputs (Pydantic / JSON schema), retrieval pipelines (ETL Experience) and end-to-end agent workflows
- Validate guardrails around tool execution: auth scoping, input/output validation, PII and prompt-injection protections, and hallucination mitigation
- Wire evaluations into CI using Dataiku Evaluations, GitHub Actions or Jenkins so every change is graded against quality, safety and cost SLOs before it ships
What you need
- Python
- SQL
- Performance testing
- LLMs
- Snowflake
- Unit testing
Good to have
- CI/CD
- ETL
- Jenkins
- Git
- Docker
- MLOps