Infrastructure Engineer

About the role

Infrastructure Engineer at AION. Observability Systems: Build and deploy comprehensive monitoring for GPU infrastructure using DCGM, NVML, and custom exporters.

What you'll do

• LGTM Stack Architecture & Operations: Deploy and manage production-scale Loki, Grafana, Tempo, and Mimir alongside Prometheus and Thanos/VictoriaMetrics
• Kubernetes Controller Development: Build custom controllers and operators for GPU workload management, scheduling, and resource allocation
• Training & Inference Monitoring: Design and implement specialized observability for AI training workloads (GPU efficiency, distributed training performance
• SLURM Integration & Monitoring: Deploy and manage SLURM clusters for HPC workloads, build observability for batch jobs, create bridges between SLURM and Kubernetes
• Multi-Tenant Monitoring Isolation: Build secure observability isolation ensuring customers access only their metrics and logs while maintaining platform-wide visibility…

What you need

• 6–10 yrs of experience
• Kubernetes
• Terraform
• Prometheus
• Grafana
• Networking
• Go
• Any graduate

Good to have

• Product strategy
• Machine learning
• Rust
• Instrumentation
• Python
• Design systems