Descriptions:
Fahd Mirza tests CatMind-1.2B, a fine-tune of the LFM 2.5 1.2B thinking model in which all reasoning traces have been replaced with irrelevant cat stories — while final answers are kept correct via a frozen LoRA adapter (rank 32). The experiment targets a fundamental question in AI research: does the actual content of a chain-of-thought trace matter, or does any sequence of tokens serve as a useful computational scratchpad?
The results are quantitatively striking. The base thinking model achieves 75.6% accuracy using genuine step-by-step reasoning. An instruct version that skips the thinking block entirely drops to 49.2%. CatMind, using cat stories in place of real reasoning, falls further to 24.3% — a 51-point collapse from swapping trace content without removing it. An empty think block performs worst at 17.2%, suggesting the cat stories at least recover a few accuracy points over pure silence.
Mirza runs the tests live on an NVIDIA RTX 6000 (48GB VRAM), including a critical ablation where he manually injects a random, externally written cat story into the think block to check whether the model secretly encodes math inside its own generated prose. The answer quality is unchanged, confirming the story text is decorative and that actual computation happens elsewhere. The takeaway for practitioners: what a reasoning model writes while it thinks is doing genuine work, and replacing it — even without removing it — causes measurable degradation.
📺 Source: Fahd Mirza · Published July 20, 2026
🏷️ Format: Benchmark Test
