Summary
Laurie Voss, head of developer relations at Arize AI and co-founder of npm, presents original benchmark research at AI Engineer tackling a question every agent builder faces but few have measured rigorously: how many instructions can a modern LLM reliably follow before accuracy degrades? The answer has practical stakes for anyone maintaining complex skills files, system prompts, or CLAUDE.md-style instruction sets.
Voss builds on IFScale, a 2025 benchmark that established a ~200-instruction ceiling for earlier models, then replicates and extends it to five current systems: GPT-4.1, Claude Sonnet 4, Opus 4.7, Gemini 2.5 Pro, and Deepseek 4. The headline finding is that the ceiling has moved by roughly an order of magnitude — but each model fails in a distinctly different and revealing way. Deepseek 4 simply forgets instructions past ~750 rules, dropping nearly half by 2,000 (the most predictable failure mode). Opus 4.7 triggers API-level safety refusals when random word combinations resemble dangerous content, requiring Voss to pre-filter words through OpenAI’s safety classifier just to get completions. Gemini 2.5 Pro holds strong to 5,000 instructions before exhausting its entire thinking-token budget and producing no output at all.
The practical takeaways cover how to structure skills files to stay within reliable instruction-following ranges, why the IFScale number is a ceiling (real instructions are harder than keyword inclusion), and what model-specific failure modes mean for evaluation strategy. Voss points to Arize AI’s open-source evaluation tooling for teams wanting to measure instruction compliance in production agents.
📺 Source: AI Engineer · Published September 09, 2026
🏷️ Format: Benchmark Test
