A randomised trial finds experienced developers slower with AI assistance
METR published a randomised controlled trial in which sixteen experienced open-source developers worked on 246 real issues in repositories they knew well — about five years of prior experience with the projects on average — using AI tools available between February and June 2025. They took 19% longer with the tools than without, and estimated afterwards that the tools had made them about 20% faster. The gap between the measured and the perceived effect is the finding that generalises furthest, because self-reported productivity gains are the evidence base for a very large share of public claims about this technology.
Observed — measured or witnessed, with the observation cited.
Subjects
- Machine-generated code · technology · not located
- METR · organisation · United States · not located
- The difficulty of evaluating machine learning systems · phenomenon · not located
Evidence
Partly verified — Core facts are sourced; some optional detail is deliberately absent.
METR — developer randomised controlled trial and task time-horizon measurements
In a randomised trial, sixteen experienced open-source developers working on 246 real issues in repositories they knew well — about five years of prior experience with the projects on average — took 19% longer to complete tasks when allowed to use AI tools available between February and June 2025, while estimating afterwards that the tools had made them about 20% faster; METR has since labelled the result historical, as not necessarily reflecting current tools or workflows. Separately, METR reports a task time-horizon metric — the human-measured duration of task a model completes autonomously with 50% reliability — as having doubled approximately every seven months over 2019–2025, measured across 170 tasks against over 800 human baselines with thirteen frontier models. V55 verification basis: metr.org was not fetched. The trial figures, the arXiv identifier 2507.09089, the task and participant counts and METR's own historical caveat were carried in search results attributing them to METR; the seven-month doubling, the 170-task suite and the 800-plus human baselines were carried with attribution to METR's March 2025 post. Any specific model's time horizon, and reports of a faster doubling in the most recent period, come from secondary summaries and are NOT asserted in this pack.
METR (Model Evaluation and Threat Research) · 2025-03 to 2026-01 · Partly verified
METR, "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity", 10 July 2025, https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/ and arXiv:2507.09089; METR, "Measuring AI Ability to Complete Long Tasks", 19 March 2025; METR, "Time Horizon 1.1", 29 January 2026
Recorded on Machine-generated code
- A human-parity claim in translation fails under document-level evaluation
- Payroll data shows a relative decline in entry-level employment in AI-exposed occupations
- An audit finds the most-cited preference leaderboard is optimised rather than merely measured
- A systematic review of 445 benchmarks reports pervasive validity failures
- A randomised trial finds experienced developers slower with AI assistance
- The Commission proposes deferring the high-risk obligations
- The second International AI Safety Report reports mixed labour findings
- The AI Index records both fast benchmark movement and doubts about benchmarks
Related
Related because they share a subject in the Atlas — never because the text looks similar.
- The second International AI Safety Report reports mixed labour findings · 2026-02
- The AI Index records both fast benchmark movement and doubts about benchmarks · 2026
- The Commission proposes deferring the high-risk obligations · 2025-11-19
- Payroll data shows a relative decline in entry-level employment in AI-exposed occupations · 2025
- An audit finds the most-cited preference leaderboard is optimised rather than merely measured · 2025-04
- A systematic review of 445 benchmarks reports pervasive validity failures · 2025-11
- A human-parity claim in translation fails under document-level evaluation · 2018