Senior QA Engineer, AI

About the role

ICE's AI Center of Excellence is looking for a Senior Software Test Engineer to own QA for our AI powered applications. This role works with Systems Analysts, Development, and QA to understand business/product and system requirements and define test scenarios for AI applications, combining core QA fundamentals, Python scripting, and API validation with specialized AI/LLM evaluation skills such as detecting hallucinations, bias, and drift.

What you'll do

• Lead test strategy for AI/ML features: model output validation, regression testing across model/prompt versions, and edge case discovery
• Build evaluation frameworks for testing AI outputs for accuracy, consistency, bias, hallucination, and safety/guardrail compliance
• Define golden datasets, ground truth labels, and release acceptance criteria with data science and ML engineering
• Automate prompt based test suites and model evaluation metrics into CI/CD release gates
• Lead and mentor a QA team, setting AI testing standards across projects

What you need

• 8–14 yrs of experience
• Test cases
• Selenium
• Python
• Regression testing
• Test automation
• SQL
• Any graduate

Good to have

• Performance testing
• Playwright
• Cypress
• Machine learning
• LLMs
• CI/CD