Descriptions:
The Latent Space podcast sits down with Philip Kiely and Ali Taha from Baseten to walk through the full lifecycle of a production inference request — from the moment a 200,000-token query arrives to the streamed response — covering prefill/decode disaggregation, KV cache-aware routing, speculative decoding with draft models, and multi-token prediction. The conversation is grounded in real production decisions Baseten has made, including which optimizations to layer in and in what order, and how to evaluate frameworks like Nvidia’s Dynamo (described as a routing and data-movement toolkit rather than an out-of-box optimizer).
A standout detail: Baseten runs an internal inference endpoint for JLM52 that its own engineers use via a Claude Code harness. That system generates GPU kernel code, profiles execution traces, identifies bottlenecks, rewrites the kernels, re-profiles, and iterates — meaning some of the GPU kernels inside Baseten’s inference engine were written by the model it runs on. This is a concrete, production example of AI-assisted infrastructure self-optimization.
The episode also covers practical guidance for companies moving from serverless inference to self-hosted GPU deployments: how to pick up open-source quantized checkpoints (avoiding the need to quantize yourself), when to use provider-trained speculative decoding models, and how PD disaggregation changes cost and latency profiles. References to KV cache offloading, GLM-52’s multi-token prediction for self-speculative decoding, and VLM-specific routing heuristics make this a strong reference episode for ML engineers working on production inference stacks.
📺 Source: Latent Space · Published August 03, 2026
🏷️ Format: Podcast
