Controlled Study: Experienced Developers 19% Slower With AI Despite Feeling 20% Faster
Summary
- • METR randomized controlled trial finds experienced developers measured 19% slower with AI tools while self-reporting 20% speed gains
- • The ~40-point perception/reality gap makes standard team velocity metrics actively misleading for AI adoption decisions
- • Effect reverses for junior developers and greenfield projects where verification overhead is lower
- • Faros AI (10k+ devs), DORA, and GitClear (200M code lines) all corroborate: more output generated, same delivery, rising churn
Details
METR RCT: 19% slower, felt 20% faster
16 experienced developers across 246 tasks showed a ~40pp gap between AI-assisted perceived and actual productivity in familiar codebases
Velocity metrics read backward
Team felt-sense of speed — the dominant input for enterprise AI adoption decisions — is actively inverted under real conditions: expert developers working in existing codebases
Faros AI: 10k+ devs, net-zero delivery
PRs merged up 98%, PR size up 150%, review time up 91%, but overall delivery essentially unchanged; 31% of PRs merged with no review at all
DORA: AI adoption drops delivery stability
Large-scale survey found higher AI tool adoption correlates with measurable drops in software delivery stability, persisting into 2025
GitClear: refactoring collapsed to <10%
Analysis of 200M changed lines shows code churn rising, copy-paste code increasing, and refactoring at a historic low — a leading indicator of rising technical debt
Code review is the new bottleneck
AI made code generation cheap but verification expensive; review volume exploded at the exact stage organizations did not re-staff, with true costs appearing downstream as incidents, churn, and burnout
METR controlled trial and large-scale industry data on AI coding tools and experienced developer productivity
What This Means
A controlled METR study, validated by multiple large-scale industry datasets, reveals that experienced developers are measurably slower with AI coding tools while genuinely believing they are faster — making the primary metric driving AI adoption decisions (team velocity perception) actively backwards. The effect is specific to experienced developers on complex existing codebases; junior developers and greenfield projects show genuine gains. This creates a fundamental ROI measurement crisis: the true costs of AI over-reliance (code churn, incidents, reviewer burnout) appear in different dashboards than the velocity metrics being celebrated. Leadership decisions to expand AI tool adoption based on developer self-reporting may be systematically miscalibrated, with the damage only visible quarters later in stability and quality metrics.
Sentiment
Skeptical of AI hype with emphasis on the perception-reality gap
“We ran a randomized controlled trial to see how much AI coding tools speed up experienced open-source developers. The results surprised us: Developers thought they were 20% faster with AI tools, but they were actually 19% slower when they had access to AI than when they didn't.”
“The most interesting finding from METR's original study was that developers thought AI was speeding them up when it was actually slowing them down. Now those same developers refuse to work without AI even for paid research. That's not necessarily evidence the tools got better. It might just mean people got dependent on something that feels productive whether or not it actually is. A 4% speedup with a range that includes zero is not the 5-10x productivity gain the industry has been promising.”
“Everything I've learned about AI coding in 2026 fits on one image. Built on METR's actual study - 16 engineers, 246 tasks, 19% slowdown measured against a 20% perceived speedup. The gap is workflow, not tools. The 14 rules close it.”
“Experienced devs using AI tools were 19% SLOWER in a controlled trial. (METR, 2025) The kicker: they predicted they'd be faster. They felt faster while being measurably slower. Measure the outcome, not the vibe.”
“The AI workflow is not 4x faster, it's ~20% slower and most people do not know it. 92% of developers use AI every day METR tested experienced engineers. They were sure they were 20% faster. The clock said 19% slower. 39 points between feeling and reality.”
Split
Perception gap (feeling faster vs being slower) vs actual measured productivity; ~60/40 split between those highlighting the irony/warning and those focused on workflow fixes.