Summary
On-policy distillation (OPD) is emerging as a key post-training technique for large language models, and this bycloud deep dive explains why. The video starts with the core problem of reinforcement learning: because RL pushes up the probability of rewarded trajectories and pushes down the rest, it sharpens a model’s distribution, making it strong at a narrow skill but less general and less diverse.
The explanation walks through how pre-training builds a broad distribution over possible next tokens, and how stacking several RL objectives into one model causes those behaviors to compete. Distillation offers a way around the issue. In the OPD setup, a copy of the model is first refined with RL to become a specialist, then used as a teacher for the original model, which receives a correction signal at every token instead of a single reward at the end.
The video describes how this lets developers train separate specialists, for example in math and code, and consolidate them into one deployable model in a modular way. It cites DeepSeek V4 and MiMo V2 Flash as examples of open research moving in this direction. It also raises the open question of which teacher a student should follow when specialists disagree, and why simply averaging their distributions can fail.
📺 Source: bycloud · Published November 18, 2026
🏷️ Format: Deep Dive







