Senior System Software Engineer - LocalAI

About the role

The LocalAI team is seeking a Senior Systems Software Engineer to develop efficient on-device AI software for RTX and DGX-class systems. The role focuses on delivering high-performance local inference with low latency, optimized memory utilization , robust infrastructure, and practical deployment on resource-constrained platforms.

What you'll do

• Build and optimize the local AI inference stack for RTX, RTX Pro, and DGX GPUs, with a focus on performance, stability
• Design and develop modern inference runtimes and execution stacks using frameworks such as llama.cpp, vLLM , PyTorch , WinML , DXCGC, and TensorRT-RTX, supporting LLM
• Perform end-to-end optimization of AI models, data pipelines, and inference runtimes to maximize performance on current and next-generation GPU architectures
• Apply model optimization techniques, including quantization, pruning, sparsity, and distillation, to enable efficient deployment of large models on local and edge devices
• Conduct system-level debugging, performance tuning, and performance-accuracy trade-off analysis; develop infrastructure for performance and accuracy sweeps

What you need

• 5–9 yrs of experience
• Data structures
• C++
• Machine learning
• PyTorch
• LLMs
• ETL
• PhD or B.Tech or B.E. or B.Sc or BCA or M.Tech or MCA

Good to have

• Deep learning