Summary
Fahd Mirza breaks down Edge Zero, a new open-source streaming inference framework that claims to run the Qwen 3.5 35B mixture-of-experts model in under 3GB of active memory — compared to the 20+ GB of VRAM normally required. The approach differs fundamentally from distillation or pruning: rather than shrinking the model, Edge Zero keeps the full 35-billion parameter checkpoint on disk as memory-mapped files and streams only the active experts needed for each token into RAM, using a trained “pre-loader” head that predicts one step ahead to overlap loading with computation.
Key benchmarked figures: 2.9GB peak memory usage, 15–18 tokens per second on a Mac Mini M4 Pro, and a quality penalty of only 3.9 average benchmark points (on HumanEval and MMLU Pro) versus the FP16 base — recovered through a technique called Recovered LoRA, which trains distillation-based adapters on top of the frozen int4 base. A smaller 8B tier built on LLaMA 3 runs at 24–25 tokens/second using roughly 1GB. The framework claims up to 59% faster decode via the pre-fetch overlap.
Currently limited to macOS with Apple Silicon via MLX (CUDA support is roadmapped), Mirza provides full installation commands and is candid about limitations: no agentic use cases, no tool use, and no production readiness. An honest early look at a genuinely novel architecture for anyone tracking edge inference and on-device AI.
📺 Source: Fahd Mirza · Published September 11, 2026
🏷️ Format: Tutorial Demo
