October 14, 2026
The Reviewer's Time Is Not in the AI Math
METR admits its new data on AI productivity is weak. The stopwatch of whoever generates ignores the time of whoever reviews the result.
METR, an organization that evaluates AI models, says AI likely speeds up developers more today than in early 2025. But by how much? The new data is, in its words, very weak evidence on the size of the increase. Between 30% and 50% of developers said they stopped submitting tasks they didn't want to do without AI, and, according to METR, the tasks with the highest expected gain dropped out of the study.
In 2025, 16 experienced developers listed real tasks, and METR randomly chose which would be done with AI. They estimated, on average, that AI had cut task time by 20%. The recorded time showed the opposite: with AI, they took 19% longer. In the screen recordings, part of the time with AI went to checking what it returned.
Larry Tesler formulated the law of conservation of complexity: every piece of software has a fixed amount of complexity, and someone always carries it. Patrick Neeman, at UX Collective, names whoever carries the checking of what AI generates: the reviewer. At METR, the person who generated with AI and the person who checked were the same, so the checking entered the recorded time.
METR itself warns that a time difference may not be a value difference, because with AI some developers began choosing a different kind of task. On a team, the time math still comes out incomplete: whoever reviews the pull request or checks the generated report is someone else, and that time doesn't enter the stopwatch of whoever generated it. Measuring the gain of an AI feature by that stopwatch alone shows one side of the work. The time to approved delivery, adding up whoever generated and whoever checked, is how the math closes.
Sources: METR (paper, arXiv) · METR (update, Feb 2026) · UX Collective, Patrick Neeman