Senior QA Engineer, AI
HyderabadFull-time8–14 yrsPosted over a month ago
Leadership QA and testing role at ICE (Intercontinental Exchange) in Hyderabad. Needs 8+ years.
About the role
This role works with Systems Analysts, Development, and QA to understand business/product and system requirements and define test scenarios for AI applications, combining core QA fundamentals, Python scripting, and API validation with specialized AI/LLM evaluation skills such as detecting hallucinations, bias, and drift.
What you'll do
- Lead test strategy for AI/ML features: model output validation, regression testing across model/prompt versions, and edge case discovery
- Build evaluation frameworks for testing AI outputs for accuracy, consistency, bias, hallucination, and safety/guardrail compliance
- Define golden datasets, ground truth labels, and release acceptance criteria with data science and ML engineering
- Automate prompt based test suites and model evaluation metrics into CI/CD release gates
- Lead and mentor a QA team, setting AI testing standards across projects
What you need
- Test cases
- Selenium
- Python
- Regression testing
- Test automation
- SQL
Good to have
- Performance testing
- Playwright
- Cypress
- Machine learning
- LLMs
- CI/CD