Summary
Fahd Mirza demonstrates running DeepSeek V4 Flash, a 284-billion parameter mixture-of-experts model, entirely on a single AMD Ryzen AI Max+ 395 APU with 120 GB of unified memory — achieving 32 tokens per second using an open-source inference engine called Loose Box.
The video explains the two key techniques behind this performance: speculative decoding via the DeepSpark draft model, which generates up to 16 candidate tokens that the main model verifies in a single fused GPU pass, and multi-token prediction (MTP) heads baked directly into DeepSeek V4 Flash’s weights. Combined, these yield a 26% throughput gain over standard autoregressive inference (25 tok/s → 32 tok/s) on consumer-grade hardware, with no cloud dependency.
Mirza walks through the complete setup — cloning and compiling Loose Box from source as a raw C++ binary over AMD’s ROCm HIP stack (no Python, no vLLM), configuring inference parameters including verification cap, KV cache settings, and expert routing (top-4 instead of default top-6), and running live queries against the served model on localhost:8000. This is a practical, reproducible reference for engineers interested in running frontier-scale open-weight models locally on AMD unified-memory hardware.
📺 Source: Fahd Mirza · Published July 27, 2026
🏷️ Format: Hands On Build
