Engineer II, QA
GurugramFull-time3–6 yrsPosted 4 months ago
Mid-level quality assurance role at Anaplan in Gurugram. Needs 3+ years.
About the role
As our AI QA Engineer, you'll develop testing strategies for full stack application testing, evaluation frameworks, and quality metrics specifically designed for LLM-powered applications. This role requires a unique blend of QA expertise, understanding of GenAI behaviour, and automation skills to ensure our AI features are reliable, accurate, and trustworthy
What you'll do
- Design and implement comprehensive testing strategies for GenAI features including conversational AI, agentic systems, and LLM-powered workflows
- Develop automated test suites for prompt testing, including regression tests that detect unintended changes in model behaviour
- Create evaluation frameworks to measure GenAI quality across multiple dimensions (accuracy, relevance, safety, consistency, latency)
- Build and maintain test datasets and golden examples that represent diverse user scenarios and edge cases
- Implement monitoring and alerting systems to detect quality degradation in production GenAI features
What you need
- Test cases
- API testing
- Performance testing
- Test automation
- Python
- Playwright
Good to have
- Machine learning
- JIRA
- Postman
- LLMs
- Prompt engineering
- CI/CD