Engineer II, QA

About the role

As our AI QA Engineer, you'll develop testing strategies for full stack application testing, evaluation frameworks, and quality metrics specifically designed for LLM-powered applications. This role requires a unique blend of QA expertise, understanding of GenAI behaviour, and automation skills to ensure our AI features are reliable, accurate, and trustworthy

What you'll do

• Design and implement comprehensive testing strategies for GenAI features including conversational AI, agentic systems, and LLM-powered workflows
• Develop automated test suites for prompt testing, including regression tests that detect unintended changes in model behaviour
• Create evaluation frameworks to measure GenAI quality across multiple dimensions (accuracy, relevance, safety, consistency, latency)
• Build and maintain test datasets and golden examples that represent diverse user scenarios and edge cases
• Implement monitoring and alerting systems to detect quality degradation in production GenAI features

What you need

• 3–6 yrs of experience
• Test cases
• API testing
• Performance testing
• Test automation
• Python
• Playwright
• Any graduate

Good to have

• Machine learning
• JIRA
• Postman
• LLMs
• Prompt engineering
• CI/CD