Senior System Software Engineer - Local AI

About the role

The LocalAI team is seeking a Senior Systems Software Engineer to lead the architecture of efficient on-device AI software for RTX and DGX-class systems. The role focuses on delivering high-performance local inference with low latency, optimized memory utilization, robust infrastructure, and practical deployment on resource-constrained platforms

What you'll do

• Build and optimize the local AI inference stack for RTX, RTX Pro, and DGX GPUs, with a focus on performance, stability
• Architect and develop modern inference runtimes and execution stacks using frameworks such as llama.cpp, vLLM, PyTorch, WinML, DXCGC, and TensorRT-RTX, supporting LLM
• Perform end-to-end optimization of AI models, data pipelines, and inference runtimes to maximize performance on current and next-generation GPU architectures
• Apply model optimization techniques, including quantization, pruning, sparsity, and distillation, to enable efficient deployment of large models on local and edge devices
• Lead system-level debugging, performance tuning, and performance-accuracy trade-off analysis; develop infrastructure for performance and accuracy sweeps

What you need

• Data structures
• C++
• Machine learning
• PyTorch
• LLMs
• ETL

Good to have

• Deep learning