Summary
Why does a coding agent go from answering in a fraction of a second to sitting silent for 19 seconds? This video from Alex Ziskind traces the slowdown to a simple cause: on every turn, the agent rereads a longer and longer conversation history, and on a single GPU that cost multiplies with each agent added. Running on an RTX Pro 6000, the first turn responds in about 0.12 seconds, while turn 25 takes 19 seconds before producing a word.
The video explains the KV cache, the stored “notes” a model keeps while reading a prompt, and why those notes compete with the model for GPU memory. It then benchmarks four setups across a 30-turn session: no cache, vLLM’s built-in prefix cache, vLLM’s CPU offload, and LMCache, a separate cache server that keeps data in system RAM and even on NVMe drives. Without a cache, the session rereads over 2 million tokens in total, while prefix caching never reads more than about 10,000 tokens per turn, roughly 14 times less.
Tests on an NVIDIA DGX Station GB300 with Qwen3 235B show cached responses around 112 to 164 milliseconds versus 1.35 seconds uncached, and a 16-client concurrent load dropping from 187 seconds to as low as 69 seconds. The key difference appears after a restart: vLLM’s own caches are lost, while LMCache keeps its cache alive and recovers instantly, though with some GPU memory release caveats. The video also touches on CacheBlend for reordered documents and a test with DeepSeek V4 Flash.
📺 Source: Alex Ziskind · Published October 13, 2026
🏷️ Format: Benchmark Test
