Summary
Optimization algorithms sit at the heart of training large language models, yet they rarely get attention. This video from bycloud explains Muon, a new optimizer that is challenging AdamW, the 12-year standard for training neural networks, using minimal math.
The video first recaps how training works: backpropagation produces gradients, and the optimizer decides how far to step. It explains how Adam uses moving averages of gradients and squared gradients, and how AdamW fixes the way weight decay is applied. It then shows how Muon differs by treating weights as matrices and orthogonalizing updates, using Newton-Schulz iteration, a fast approximation built from repeated matrix multiplications that suits GPUs.
The main practical hurdle is distributed training, because Muon needs the whole matrix while large matrices are split across many GPUs, adding communication overhead. The video notes that Kimi and DeepSeek have adopted it, and that Kimi’s work on making Muon scalable identified fixes such as weight decay. The creator also shares a quick experiment comparing AdamW and Muon on small transformer models.
📺 Source: bycloud · Published October 24, 2026
🏷️ Format: Deep Dive
