AI QA Engineer

About the role

AI QA Engineer at apexanalytix. Own the test strategy for our AI products, covering LLM features, agentic workflows, RAG pipelines, and ML models.

What you'll do

• Own the test strategy for our AI products, covering LLM features, agentic workflows, RAG pipelines, and ML models
• Write and execute test plans, cases, and reports; manage the defect lifecycle; and perform exploratory testing of AI features
• Define acceptance criteria and quality metrics for non-deterministic outputs (accuracy, groundedness, hallucination rate, relevance
• Build and maintain golden datasets, ground-truth labels, and regression suites with subject-matter experts, reflecting real scenarios and edge cases
• Develop automated evaluation pipelines (DeepEval, Ragas, promptfoo, LangSmith) that run in CI/CD on every prompt, model, retrieval, or code change

What you need

• 2–3 yrs of experience
• Test cases
• API testing
• Selenium
• Test automation
• Unit testing
• SQL
• Any graduate

Good to have

• Python
• Playwright
• LLMs
• Machine learning
• CI/CD
• ETL