Descriptions:
Joel Becker, a researcher at METR (Model Evaluation and Threat Research), presents two empirical studies that together expose a striking gap between AI benchmark performance and real-world productivity outcomes. The first study introduces METR’s “time horizon” benchmark — measuring autonomous task completion time across software engineering, ML research, and cybersecurity tasks — and documents that frontier model capability on this metric has been doubling approximately every six to seven months.
The second study takes a fundamentally different approach: a randomized controlled trial involving 16 experienced developers working on 16 large, mature open-source repositories. Tasks were randomly assigned to “AI allowed” or “AI disallowed” (no LLMs, no AI autocomplete — software development circa 2019) conditions. Despite benchmark performance implying substantial AI capability, the measured productivity gains in this field experiment were considerably more modest, raising hard questions about external validity of controlled evaluations.
Becker dissects the limitations of both methodologies. Benchmarks saturate quickly and use small, contained, synthetic problems that lack the messiness of real codebases. Field experiments have better external validity but are harder to scale and maintain signal as models improve. The talk is essential viewing for researchers, enterprise AI buyers, and anyone trying to reconcile headline benchmark numbers with the actual productivity evidence emerging from rigorous empirical work.
📺 Source: AI Engineer · Published December 24, 2025
🏷️ Format: Deep Dive
