Descriptions:
Quantize a single number in a model and it gets 20% dumber. That finding, from the super weights paper, is why Daniel Han’s claim is less absurd than it sounds: GLM 5.2 goes from 1.5 terabytes to 250 GB, 86% smaller, without being 86% dumber. Layers are wildly unequal. The first and last carry enormous weight, the middle ones barely matter, and a model trained on 30 trillion tokens never saturates its parameters, so many sit near zero and can simply be set there. Choosing which layers stay in high precision is a combinatorial search, not a setting.
NVFP4 is the format the NVIDIA side leans on, a 4 bit float where every group of 16 values shares one FP8 scale, targeting under 1% accuracy loss. New architectures keep breaking the old heuristics: quantize the linear attention layers and the model looks fine right up until a long context turns it to gibberish. Post training quantization works out of the box above roughly 20 billion parameters and needs quantization aware distillation below that. The panel is blunt that benchmarks only cover verifiable tasks, so the real test is running the model in an actual harness, and Han’s preferred signal is KL divergence between the BF16 and quantized output logits rather than any accuracy score.
Speaker info:
Chris Alexiuk, moderator (NVIDIA):
– https://x.com/llm_wizard
– https://www.linkedin.com/in/csalexiuk
Daniel Han (Unsloth):
– https://x.com/danielhanchen
– https://unsloth.ai
Asma Beevi (NVIDIA):
– https://www.linkedin.com/in/asma-beevi-k-t-433053a2
– https://realasma.github.io
Merve Noyan (Hugging Face):
– https://x.com/mervenoyann
– https://hf.co/merve
Parth Sareen (Ollama):
– https://github.com/parthsareen
– https://parthsareen.com
Timestamps:
0:00 – Welcome and the panel
0:53 – What compression means to each of them
3:05 – GLM 5.2 from 1.5 terabytes to 250 GB
4:08 – When each of them got the compression bug
8:19 – QLoRA and finetuning on a T4
11:44 – 86% smaller without being 86% dumber
12:46 – Why layer importance is so uneven
14:26 – The super weight: one number, 20% dumber
14:51 – Evaluating the quantized checkpoints
16:37 – What NVFP4 actually is
17:55 – Does compression matter beyond the toaster
21:49 – Why compress a big model instead of using a small one
24:30 – Where Ollama fits
28:54 – How hard NVFP4 is to produce
32:46 – The cursed era of model architectures
35:17 – Why linear attention layers break quantization
37:22 – Where compression goes next
43:22 – How do you know a quant is any good
