Summary
Fahad Mirza benchmarks the Qwen 3.8 4B distilled model — built by the Emperor team, not Alibaba — across three quantization levels (Q4KM, Q6K, Q8) running on a single consumer GPU via llama.cpp on Ubuntu. The model itself is notable: a full-parameter distillation from the 2.4 trillion-parameter Qwen 3 giant, trained on 45,000 curated reasoning traces to compress its chain-of-thought behavior into a sub-3GB package.
The benchmark uses a structured prompt — a pandemic ethical dilemma with no factual answer — specifically chosen to expose reasoning depth differences that simple Q&A prompts obscure. Measured metrics include time to first token, token generation speed, total wall-clock time, thinking depth (word count before final answer), and output length. Results: Q4KM wins on speed (143.5 tok/s, 3.2s TTFT) and fits on any GPU above 4GB VRAM at 2.78GB. Q6K produces the shortest response and fastest total time (5.38s) because it stops earlier. Q8 is slowest (9.2s TTFT) but shows nearly double the thinking depth at 587 words versus 311 for Q4, suggesting higher precision unlocks more deliberate reasoning before committing.
The key finding is that all three quantization levels reached the same ethical conclusion — destroy the sample to save 200 million lives — indicating that the core reasoning survived compression intact across the entire Q4-to-Q8 range. Mirza recommends Q4KM as the daily driver for most users, with Q8 worth the overhead for tasks that reward deeper reasoning.
📺 Source: Fahd Mirza · Published August 22, 2026
🏷️ Format: Benchmark Test
