Summary
NVIDIA’s Nemotron Lab 3 Puzzle 75B compresses the 120B-parameter Nemotron Super model down to 75B total parameters — with only 9.3B active per token — while preserving accuracy through a four-stage PUZZLE compression pipeline. Fahd Mirza walks through a full deployment on a single NVIDIA H100 (80GB VRAM), from downloading the model off Hugging Face to serving it via vLLM with speculative decoding enabled via multi-token prediction (MTP).
The video explains each compression stage in accessible terms: iterative pruning with knowledge distillation (repeated three times to account for shifting layer importance), long-context distillation at 128K and 512K token windows, reinforcement learning to recover coding and agentic capabilities most damaged by pruning, and FP8 quantization with MTP heads baked directly into the weights. The hybrid architecture — combining Mamba, mixture-of-experts, and attention layers — delivers roughly 2x throughput versus a standard transformer and scales from one to eight concurrent requests at 1M context on a single H100.
Live coding tests include a merge-intervals function and a complex HTML canvas animation rendering a physics-accurate rotating trompo — both handled cleanly by the compressed model. Measured throughput runs around 11.7 streaming chunks per second, with caveats around enforce-eager mode inflating time-to-first-token. The video is a practical resource for ML engineers evaluating single-GPU deployment of large-scale models or interested in structured pruning and distillation workflows.
📺 Source: Fahd Mirza · Published July 09, 2026
🏷️ Format: Hands On Build
