Inference at Scale: Deploying Open-Source LLMs in Production with vLLM and llm-d

Training gets the headlines, but inference is where generative AI meets production — and where the costs, latency, and scaling challenges actually live. This session demystifies LLM inference from the ground up: how models generate tokens, why the KV cache and the prefill/decode split matter, and what an inference server actually does to make GPUs go "brrrr." From there we turn to the hard part — running inference at scale. As reasoning and agentic workloads drive token demand 20x higher (and compute 150x higher), single-node serving stops being enough.

We'll show how Red Hat approaches distributed inference with llm-d, a Kubernetes-native, open-source initiative built jointly with Google, NVIDIA, Hugging Face, and others. You'll see how disaggregated prefill and decode, intelligent request scheduling, KV-cache-aware routing, and hardware-agnostic serving across NVIDIA, AMD, Intel, and Google TPUs come together to maximize GPU utilization and hit demanding SLOs — across edge, private, and public cloud.

We'll connect the dots from vLLM as the de facto inference engine to llm-d as the scaling layer, rounding out the picture with validated and quantized models that stretch more tokens out of fixed hardware.

Attendees will leave understanding not just what inference is, but how to deploy it efficiently and reliably at enterprise scale.