AI QA Engineer

About the role

The AI Test Engineer is responsible for validating the quality, reliability, safety, performance, and governance of AI systems, including LLMs, RAG pipelines, and AI agents. Unlike traditional QA role, this position focuses on LLM evaluation, behavioral testing, hallucination, bias detection, AI test automation, and alignment with Responsible AI governance standards.

What you'll do

• Define AI test strategy for LLMs, AI agents, and orchestration workflows leveraging Tools & framework like DeepEval, Arize
• Design and maintain golden evaluation datasets, prompt test cases, and benchmarking frameworks for LLMs, RAG systems
• Perform regression testing across prompt updates, agent logic changes, multi-step agent workflows and model upgrades
• Measure and analyse accuracy, consistency, latency, throughput, and cost efficiency of AI services
• Integrate AI test suites into CI/CD pipelines with automated quality gates

What you need

• 3–6 years in quality assurance & testing
• Test automation
• Selenium
• Unit testing
• Python
• Java
• Playwright
• Any graduate

Good to have

• LLMs
• CI/CD
• Test cases
• Azure
• Machine learning
• Regression testing