Every AI coding tool ships with a productivity number. The trouble is that most of those numbers come from surveys, vendor telemetry, or contrived tasks. Two controlled experiments now exist, and they disagree by a wide margin: one found developers 55.8% faster, the other found them 19% slower. Both are real results. The difference between them is the most useful thing the research has to say about whether AI will speed up your work.
The vendor experiment: 55.8% faster
The first controlled study came from Microsoft and GitHub researchers with an MIT co-author in 2023. Recruited software developers were asked to implement an HTTP server in JavaScript, a well-defined, self-contained task, with half given GitHub Copilot and half not. The group with the AI pair programmer completed the task 55.8% faster than the control group.
That is a large effect, and it is probably true for that kind of task: a greenfield, standard problem with no existing codebase to understand, done by developers who did not need to maintain the result. The authors were employees of the company selling the tool, which is a reason for caution about the framing rather than the measurement.
The independent experiment: 19% slower
In 2025 the research nonprofit METR ran a randomised controlled trial by having 16 developers complete 246 tasks on well-known open-source repositories they regularly contributed to, with an average of five years of experience on those projects. Each task was randomly assigned to allow or forbid AI tools; when allowed, developers mostly used Cursor with Claude models.
The result went against everyone’s expectation, including the researchers’. Before starting, developers forecast that allowing AI will reduce completion time by 24%. Afterwards, they estimated it had sped them up by about 20%. On the measured clock, they took 19% longer to complete issues when AI was allowed.
Why the two studies disagree
The two experiments differ on the dimensions that plausibly matter most.
Familiarity with the codebase. The METR developers were experts in their own repositories. Much of what an AI assistant offers, such as explaining unfamiliar code or recalling an API, is worth less to someone who already knows the code intimately. The Copilot task was fresh code with no context to learn.
Task type. Implementing a standard server is close to the training distribution of every coding model. Fixing a real issue in a mature project with its own conventions, tests and edge cases is not.
Quality bar. Open-source maintainers ship code they will have to live with and that reviewers will scrutinise. Time spent reviewing, correcting and cleaning up AI output counts against the tool in that setting; in a one-off benchmark task it does not.
Neither study is wrong. They measure different jobs. The mistake is applying either number to the other job.
The perception gap
The most important METR finding is not the 19%. It is that developers believed the opposite of what the stopwatch showed, before and after. A 24% expected speed-up, a 20% perceived speed-up, and a 19% measured slowdown, in the same people on the same tasks.
That has a direct implication for anyone evaluating these tools: your sense of whether AI is making you faster is not reliable evidence. Self-reported productivity, which is what most surveys and most team retrospectives run on, was wrong in the one setting where it could be checked against a clock.
What the evidence does not cover
The METR study was 16 developers, early-2025 tools, and one setting. The authors describe it as a snapshot and say it does not imply the tools are useless elsewhere. Tools have changed since; METR has said it published further data on later tools in 2026. The Copilot study was one task in one language. Neither says anything about junior developers, unfamiliar codebases, or long-term skill effects, all of which are separate questions with separate literatures.
What this means in practice
If you are working in code you know well, with a high quality bar, the best available controlled evidence says AI assistance may cost you time, and that you will not notice. If you are starting something from scratch, or working in an unfamiliar language or library, the evidence says the gain can be large. The honest approach is to measure rather than feel: time a few real tasks with and without the tool, because the one trial that did so found feelings pointed the wrong way.
Common questions
Does AI make programmers more productive?
It depends on the task. A controlled study of a standard greenfield task found Copilot users 55.8% faster. A randomised trial of experienced developers on their own mature projects found AI tools made them 19% slower.
Why did the METR study find AI slowed developers down?
The participants were experts in their own repositories, so the assistant’s main advantages mattered less, and time spent reviewing and fixing AI output counted against it. The study measured real issues to a maintainer’s quality bar.
Do developers know when AI is helping them?
Not reliably. In the METR trial, developers expected a 24% speed-up, believed afterwards they had got about 20%, and were measured at 19% slower.
Is the METR result still true with newer tools?
Unknown from that study, which the authors call a snapshot of early-2025 tools. The methodological point, that perceived speed-up and measured speed-up can diverge sharply, does not depend on the model version.
The bottom line
AI coding tools can make you much faster on fresh, standard work and can make you slower on expert work in a codebase you know, and in the latter case you will probably feel faster anyway. Measure it on your own tasks before you believe either number.
Did this answer your question?
Thanks. That helps us decide what to fix.
Leave a comment