Summary
Fahd Mirza makes the case that open-source local AI is far from dead — it has simply shifted from competing on parameter counts to competing on compression engineering. The video surveys three recent developments that together signal a coordinated push: Colibri, a GLM 5.2 quantized to int4 with the multi-token prediction head deliberately kept at int8 to preserve speculative decoding performance, and an expert-caching engine that uses system RAM as a warm cache to sustain throughput.
Prism ML’s Bonsai 27B takes a different route: fully ternary quantization where every weight — including embeddings, attention, MLP, and output head — is reduced to -1, 0, or +1, with each 128-weight block sharing one full-precision scale factor. This approach reaches 1.71 bits per weight, shrinking a 27B model from 54 GB to under 6 GB, with a 1-bit variant that runs on an iPhone. Tencent’s Hy3 1-bit meanwhile brings a 295-billion-parameter flagship model to roughly 83 GB on a single GPU through selective quantization: attention layers and embeddings stay at 6–8 bit, shared experts at 5–6 bit, and only the routed experts — the bulk of the file — take the aggressive 1-bit compression, with per-layer tuning on top.
The thread tying these together, Mirza argues, is Tencent’s AngelSlim toolkit: an open-source compression infrastructure covering quantization, speculative sampling, sparse attention, distillation, and multimodal compression across DeepSeek, Qwen, and other major model families — a sign that compression is becoming permanent infrastructure, not a one-off launch trick.
📺 Source: Fahd Mirza · Published July 16, 2026
🏷️ Format: Opinion Editorial
