Machine-generated code

technology

Writing and modifying software with model assistance — the most heavily instrumented deployment of language models, and the one where measured benchmark movement and measured human productivity most conspicuously fail to line up. On SWE-bench Verified, a 500-task set of real GitHub issues human-validated by 93 developers and released with OpenAI in August 2024, the 2026 AI Index reports resolution rates rising from around 60% to near 100% within a single year. Against that, a randomised controlled trial published by METR in July 2025 found that sixteen experienced open-source developers working on 246 issues in repositories they already knew took 19% longer with early-2025 AI tools than without, while estimating afterwards that the tools had made them about 20% faster. Both results are real measurements of different things: one of whether a patch passes a held-out test suite, one of how long a familiar expert takes on a familiar codebase. Neither licenses a claim about software employment, and the gap between the perceived and measured speed-up in the trial is itself a finding worth more than either number.

A technology is not a place. Drawing it on a map would assert something about the world that no stored fact supports.