Track Manager - AWS IAC, Terraform,Python

About the role

Production Eval & Observability Engineer Experience Level 6–9 yrs Senior Developer Role Summary Builds syf-online-eval: production-sampling sidecar closing the loop with syf-eval — sampling live sessions, running async LLM-judge scoring, and surfacing drift/regression on a dashboard.

What you'll do

• Build the sampling sidecar consuming OTel gen_ai.* traces from production agents, with async judge scoring
• Build drift detection against each agent\'s golden-set baseline
• Build the dashboard (usage, pass-rate by behavior-tag, drift alerts, cost)
• Respect data-classification gating on sampled prompt/completion content (capture default OFF)

What you need

• 6–9 yrs of experience
• AWS
• Terraform
• Python
• Splunk
• Grafana
• LLMs
• Any graduate