Descriptions:
At the AI Engineer conference, Red Hat’s Yuchen Fama (vLLM contributor and product manager) and Ashish Kamra (senior manager of performance engineering) deliver a deep technical session on why agentic workloads break classic LLM serving assumptions — and two architectural techniques that address the problem: KV cache-aware routing and prefill/decode (P/D) disaggregation.
The presenters draw on real-world agentic traces, including SWE-bench and Claude Code sessions, to show that agentic workloads feature up to 3,000-turn conversations, KV cache hit rates exceeding 90%, and input/output ratios over 100:1. These patterns make steady-state throughput benchmarks misleading and create volatile cache management, frequent evictions, and scheduling complexity that require purpose-built infrastructure responses. KV cache-aware routing addresses time-to-first-token (TTFT) by directing requests to the GPU instance already holding the relevant cached prefix. P/D disaggregation, implemented in LLMD — Red Hat’s CNCF-hosted Kubernetes-native inference framework — separates prefill pods (compute-heavy, bursty) from decode pods (memory-bandwidth-hungry, latency-sensitive), preventing long prompt processing from stalling token streaming.
The talk includes a live demo of prefix routing reducing TTFT to approximately one second, a case study on the open-source coding model GLM 5.2, and references Red Hat’s position as the top contributor to vLLM, LLMD, and the lm-eval, LLM Compressor, and Speculators open-source projects. A free distributed inference course on DeepLearning.AI with Andrew Ng is also mentioned.
📺 Source: AI Engineer · Published August 27, 2026
🏷️ Format: Deep Dive
