Senior QA Engineer - AI

About the role

The same input produces different output on every run, an incorrect answer can be entirely fluent, and a prompt change in one flow can degrade another without any visible signal. The core of this role is establishing what correct means for these systems, building the datasets and scoring that measure it, and operating the release gate that prevents a regression from reaching customers.

What you'll do

• Build and maintain golden datasets for our AI products, trace mined from production or generated, and verified against a documented source of truth
• Design layered scoring for AI outputs: deterministic rules where behaviour can be asserted, and model-graded evaluation where it cannot
• Manage evaluation runs end to end: schedule and execute them against release candidates, compare results against prior baselines
• Own and operate the release gate that determines whether a prompt or model change is approved for production
• Identify failure modes ahead of customers, including hallucination, incorrect tool selection, loss of context across conversation turns, and PII exposure

What you need

• API testing
• Python
• Test cases
• PRD writing

Good to have

• Capital markets