Summary
Fahd Mirza demonstrates running GLM 5.2 — a 744 billion parameter mixture-of-experts model from Tsinghua’s KEG Lab — on a single consumer machine using Colibrì, a minimal inference engine written in a single C file with zero runtime dependencies. The trick is that MoE models only activate a small fraction of their parameters per token: GLM 5.2 selects 8 experts out of 256 at each layer, meaning the model’s full 370 GB weight file never needs to reside in RAM at once.
Colibrì exploits this by keeping the ~10 GB dense attention layers permanently in RAM, then streaming expert weights from disk on demand. A caching layer learns which experts are accessed most frequently and pins them in memory — roughly 100 GB of the most-used experts stay hot, while the rest stream from disk. This makes the first few tokens slow (cold cache) but throughput improves as the session continues. Crucially, the GPU sits completely idle throughout: with 132 GB of RAM on a 20-core machine, inference is entirely CPU and disk bound.
Mirza walks through the full setup: downloading the model from Hugging Face, building Colibrì with OpenMP, running the memory blueprint command to preview the RAM/disk/GPU split before any inference, and enabling speculative decoding for additional speed. The video is a practical guide for anyone curious whether truly massive open-weight models can be run without a GPU cluster, and what the realistic performance and bottleneck profile looks like on commodity hardware.
📺 Source: Fahd Mirza · Published July 14, 2026
🏷️ Format: Hands On Build
