Summary
Language models are trained once and then frozen, leaving them as snapshots of a changing world. This video from bycloud examines why that is a problem and whether giving models something like sleep could help. It opens with a benchmark showing that the knowledge cutoffs labs claim for their models often differ sharply from what testing reveals.
The explainer separates two failure modes of continual learning: catastrophic forgetting, where new learning damages old capabilities, and plasticity loss, where a model gradually becomes worse at learning anything new. It also explains why retrieval and long context windows are not a true substitute for learning. A paper from Zephra studies plasticity loss in GPT-style transformers from 5 million to 314 million parameters trained across a sequence of eight languages.
The second half covers a hybrid attention and Gated DeltaNet (GDN) design in which the model replays a chunk of context several times, updates its fast weights, and then discards the KV cache. Results on Rule 110 simulation tasks and the GSM Infinite long-context math benchmark show accuracy rising with the number of sleep loops, even though memory size and the single forward pass at answer time stay the same.
📺 Source: bycloud · Published November 10, 2026
🏷️ Format: Deep Dive
