Descriptions:
Justin Reock, head of developer relations at DX (recently acquired by Atlassian), uses the AI Engineer conference to cut through conflicting AI productivity narratives with real data — and the findings are more nuanced than most vendor claims suggest.
The talk opens with a striking contrast: Google reports a 10% productivity boost from AI coding tools, while the widely-cited METR study found a 19% decrease in measurable output — even as every developer who participated felt more productive. DX’s own aggregate data shows modest average gains (2.6% increase in change confidence, 3.4% improvement in code quality, 1% reduction in change failure rate), but these averages mask dramatic volatility. Breaking the same data down by company reveals swings of plus or minus 20% across identical metrics, with some organizations effectively shipping 50% more defects than before AI adoption. Reock identifies the common failure modes: top-down mandates, lack of structured enablement, and over-reliance on utilization metrics that measure technology consumption rather than engineering outcomes.
The second half introduces DX’s DXAI measurement framework, combining three data types — API telemetry, experience sampling added directly to PR workflows, and high-participation survey data. Reock argues that foundational developer experience metrics like change failure rate remain more reliable indicators of AI value than acceptance rate or daily active usage numbers, and that organizations succeeding with AI adoption are those treating developer experience as a systems problem rather than an individual behavior problem.
📺 Source: AI Engineer · Published December 19, 2025
🏷️ Format: Deep Dive
