First of five on what the 2026 evidence says once you read past the announcement.
What follows: why four of the five agent containment failures needed no novel exploit, what actually paced the frontier, where the AI bill goes next, and what to check before you run an open weight model.
There is a failure mode in AI-assisted development that does not look like failure. The code compiles and the test passes. The diff is small and it addresses the ticket. Review signs off because nothing is obviously wrong to point at, and the change ships with the original problem still inside it.
When the security research team at 1Password ran 6,080 patch attempts against real disclosed CVEs, 37.5 percent of the patches that did work came back fragile, meaning they closed the specific path someone had tested and left the underlying cause untouched. A further 4.5 percent introduced a vulnerability that had not been there before. Neither kind announces itself. They pass the check, close the ticket, and convert a known issue into a resolved one on somebody’s dashboard.
All of which depends on review catching it, and review is where the second half of the problem lives. Across 22,000 developers, the teams that adopted AI most heavily saw median pull request review time rise 441 percent and the share of pull requests merged with no review at all rise 31 percent. The work did not disappear. It landed on the one control that was already carrying the most weight.
The rest of this piece is the other half of that trade. The AI coding productivity data is better than the backlash suggests and worse than the slide decks claim, and the two figures everyone quotes at each other are both correct.
Search for whether AI makes developers faster and the results will hand you a study that found them 19 percent slower. One headline I pulled this week still calls it the AI coding productivity myth. The study is real, it was careful, and the people who ran it revised the number seven months ago. Almost nobody quoting it has noticed.
I use these tools every day and I am not interested in talking anyone out of them. What I want is for the AI coding productivity data to be read correctly, because the two loudest figures in the field, a 55.8 percent speedup and a 19 percent slowdown, are describing the same tool handed to different people working on different code.
The number everyone is still quoting
METR ran a randomized controlled trial in 2025. Sixteen developers, 246 tasks, in repositories averaging over 22,000 GitHub stars and more than a million lines of code. The developers averaged five years on the specific repository they were working in, with around 1,500 commits each. Every issue was randomly assigned to allow or forbid AI, and the developers were paid $150 an hour, sometimes to work without it.
The finding: allowing AI increased completion time by 19 percent. Before starting, those developers predicted AI would cut their time by 24 percent. After finishing, having actually done the work, they still estimated they had been 20 percent faster.
How to read the signs, because this is where people go wrong
The 2025 paper measures completion time. A positive number means the task took longer. So +19 percent means slower.
METR’s 2026 update measures speedup. A negative number means slower. So -18 percent means slower.
Those two figures point the same direction. The original group’s result did not reverse between papers. The axis flipped, and the sign flipped with it. Anyone comparing +19 against -18 and concluding the finding changed has read the labels rather than the measurement.
Here is the February 2026 update in full. METR reran the experiment with 57 developers, 143 repositories and over 800 tasks. The original cohort came back at -18 percent, with a confidence interval running from -38 percent to +9 percent. The newly recruited developers came in at -4 percent, interval -15 percent to +9 percent.
That -4 percent is the genuinely new result, and it is a fifth the size of the headline everyone repeats.
Now the intervals, which are the part that almost never survives into coverage. A confidence interval is the range the true value plausibly sits in. When that range crosses zero, the study cannot rule out the opposite conclusion. Both of these cross zero. The data is consistent with AI having made those developers modestly slower, and it is also consistent with AI having made them up to 9 percent faster.
METR says so themselves, and adds a problem with their own design: developers are declining to take part rather than work without AI, which drags their estimate down. Their words are that the new data “is only very weak evidence” and that “the true speedup could be much higher.”
What the AI coding productivity data shows once you sort it
The strongest study in the field points the other way, and it is bigger than METR’s by two orders of magnitude.
Cui, Demirer and colleagues ran three randomized trials across roughly 5,120 developers at Microsoft, Accenture and an unnamed Fortune 100 company, published in Management Science. Developers using GitHub Copilot completed 26 percent more tasks per week. At Microsoft, pull request approval rates went up about 10 percent, and the authors found no evidence that code quality fell.
Google ran its own trial with 96 engineers on an enterprise-grade task and measured roughly a 21 percent reduction in completion time, while noting that “our confidence interval is large.”
Then there is the number you see on every vendor slide. A 2023 GitHub and Microsoft experiment found developers 55.8 percent faster. That one deserves a closer look: 70 completed sessions, participants recruited from Upwork, and the task was writing an HTTP server in JavaScript as fast as possible. The confidence interval runs from 21 percent to 89 percent. Two of the four authors worked for the company selling the product, and they state plainly that the study “does not examine the effects of AI on code quality.”
Against all of those, DX tracked pull request throughput against AI usage across 400 companies from November 2024 to February 2026 and found an increase of 7.76 percent. They excluded teams that set PR targets for individual engineers, so the figure is not inflated by people gaming a metric.
The benchmarks are the weakest evidence in the room
Model release posts lead with SWE-bench scores, and that benchmark has been taken apart twice by people who went and checked.
York University researchers found that 32.67 percent of successful patches had the solution sitting in the issue report or its comments, and another 31.08 percent passed only because the test cases were too weak to catch a wrong answer. Filtering both out dropped the resolution rate from 12.47 percent to 3.97 percent.
A peer-reviewed paper at ICSE 2026 went further and ran every repository test rather than only the ones the pull request touched. Of 877 patches that looked correct, 7.8 percent failed the full suite outright, and differential testing found 29.6 percent behaving differently from the reference fix. The authors estimate about 11 percent of apparently-passing patches are simply wrong, inflating reported scores by 6.4 percentage points.
Stack those filters in order and a headline benchmark number bears little relation to whether the change would survive your pipeline.
The studies agree more than they look
Line those results up by how well the developer knew the code in front of them.
Upwork freelancers on a greenfield task, seeing the codebase for the first time, came in 55.8 percent faster. Junior developers in the Management Science trials gained 21 to 40 percent. Senior developers in those same trials gained 7 to 16 percent, and the authors separately found short-tenure workers gaining more than long-tenure ones inside the same companies. METR’s participants, five years and 1,500 commits into a million-line repository, landed somewhere between slightly slower and slightly faster, with the interval crossing zero.
The benefit tracks unfamiliarity. It is largest where the developer knows least about the specific code, and it shrinks toward nothing as their knowledge of that code goes up. Those studies are not in conflict. They are four points on one line, and the line has a slope.
This lands differently depending on where your teams sit. Onboarding someone into an unfamiliar stack, standing up something new, working in a language a person uses twice a year, all of that is where the gain lives. A principal engineer who has owned a subsystem for six years is the case where the tool has the least room to help and the most room to interrupt.
DORA’s 2025 report put the same idea in one sentence: “AI doesn’t fix a team; it amplifies what’s already there.” Justin Reock at DX, explaining his 7.76 percent, put it another way. Writing code was never the bottleneck. Planning, scoping, review and handoffs are where the time goes, and none of those got faster.
Where the cost lands
Faros AI has the best quality dataset I found. Twenty-two thousand developers across more than 4,000 teams, two years of telemetry, comparing each organization’s two lowest-adoption quarters against its two highest. Throughput rose. So did everything else.
| Measure | Low adoption to high adoption |
|---|---|
| Task throughput per developer | +33.7% |
| Epics completed per developer | +66.2% |
| Bugs per developer | +54% |
| Incidents per PR | +242.7% |
| Average PR size | +51.3% |
| Median PR review time | +441.5% |
| PRs merged without review | +31.3% |
| Code churn | +861% |
Read the throughput and the incident rate together rather than separately. The delivery gain is real and so is the defect cost, in the same dataset, measured the same way. Faros is careful about its own limits: this is observational, and an organization’s high-adoption quarters are also its later quarters, so some of this is whatever else happened during that time.
GitClear analysed 623 million code changes and found block duplication up 81 percent since 2023, while refactoring fell from 21 percent of all changes in 2022 to 3.8 percent so far in 2026. DORA’s throughput finding flipped from negative to positive between its 2024 and 2025 reports, and in both years the relationship between AI adoption and delivery stability stayed negative.
Sonar surveyed 1,149 professional developers. Ninety-six percent do not fully trust that AI-generated code is functionally correct. Eighty-eight percent report at least one negative effect on their technical debt. Thirty-eight percent say reviewing AI-generated code takes more effort than reviewing a colleague’s.
Two older results point the same way and should be read with their dates attached. Uplevel compared about 800 developers with and without Copilot access, found no significant change in efficiency, and measured 41 percent more bugs in the group using it. That study ran on completion-style tooling rather than agents, so treat it as a floor rather than a current reading. At the firm level, an NBER working paper surveying nearly 6,000 senior executives found nine in ten reporting no measurable productivity change over three years, though it proxies productivity as sales per employee across the whole economy, which is a blunt instrument for a software question.
Put that next to the 441 percent jump in median review time and the 31 percent rise in unreviewed merges, and the shape of the problem is legible. The work did not disappear. It moved downstream to the reviewer, and in a measurable share of cases the reviewer stopped doing it.
Security is the flat line
Veracode has tested more than 150 models over two years on the same 80 coding tasks across four languages. The share producing secure code sits at about 55 percent. Their summary: “Two years of ‘revolutionary’ model releases have moved the security needle from approximately 55% to… approximately 55%.”
The breakdown matters more than the average. SQL injection gets handled correctly 82 percent of the time and insecure cryptography 86 percent. Cross-site scripting gets handled 15 percent of the time, and log injection 13 percent. By language, Python passes 62 percent, C# 58, JavaScript 57, Java 29. Reasoning-focused models reach 70 to 72 percent, the best of any category, which Veracode still calls far from acceptable for production.
Remediation is worse than generation. Off-by-1 Labs, the security research team at 1Password, ran 6,080 patch attempts from Claude Opus 4.8 and ChatGPT 5.5 against six recently disclosed CVEs, with both models running under elevated cyber access programs. Twenty-six percent produced a complete fix that did not change application behaviour. Fifty-one percent failed to fix the bug at all. Four and a half percent introduced a new vulnerability. Among the patches that did work, 37.5 percent were judged fragile, closing the tested path while leaving the root cause in place.
That gap between what these models generate and what they can safely repair is where the next piece in this series picks up. It also connects directly to the advisory volume problem I wrote about recently, since a patch that closes one path and leaves the root cause is a patch your scanner will call green.
Adoption is climbing and so is the bill
Nothing in the quality data has slowed uptake. DORA put AI use at 90 percent of respondents in 2025. DX measures 92.6 percent monthly adoption across 121,000 developers, with the AI-generated share of code in their sample going from 34 percent in the first quarter of 2026 to 52 percent in the second. Google says about 75 percent of its new code is now AI-generated, up from roughly 25 percent in late 2024, on a metric the company has never defined as lines, characters or acceptance rate.
Developer sentiment has not tracked the adoption curve. Stack Overflow’s 2025 survey found 84 percent using or planning to use these tools, while 46 percent actively distrust their accuracy against 33 percent who trust it, and 3 percent trust it highly. The most common complaint, from 66 percent of respondents, is code that is “almost right, but not quite.” Forty-five percent say debugging AI-generated code takes longer than debugging their own.
Where companies are pulling back, the reason is the invoice. Samsara capped usage for non-technical staff and expanded it for engineering. Docusign retuned its agents to pull only relevant context and cut token consumption roughly in half. Compass put per-engineer budgets on AI tool spend. Microsoft moved its Experiences and Devices engineers off Claude Code licenses to Copilot CLI over token billing. Median quarterly AI spend per organization in DX’s sample went from about $1,500 to about $44,000 in four quarters, and Gartner expects AI coding costs to exceed the average developer salary by 2028.
Exactly one quality-driven rollback shows up in credible reporting, from a Pragmatic Engineer survey, where an engineering lead at a company of more than 10,000 people said they “have now rolled back some of our AI tools to deal with the drop in quality.”
The return figure you will see quoted against those costs deserves a label. DORA’s 2026 report puts first-year ROI at 39 percent with an eight-month payback, and that number is the output of a financial model built on assumed salary, task mix and downtime costs, not a measurement of anything. The same model raises change failure rate from 5 to 6 percent after adoption, which is the part vendors tend to leave off the slide.
What I would measure instead
Four things, if I were setting this up for a team today.
Track review time and unreviewed merges before you track throughput. Throughput will go up. The Faros numbers say the constraint moves to review, and review is the control that catches everything the model got almost right.
Track incidents per pull request rather than bugs found. Bugs found is a measure of how hard you looked. Incidents per PR is a measure of what reached production, and it is the number that moved 242 percent.
Know where your codebase familiarity sits. If most of your work is people in code they know well, expect the DX number rather than the vendor number, and plan capacity on 8 percent rather than 50. If you are onboarding heavily or working across unfamiliar stacks, the larger figures are genuinely available to you.
Stop measuring the percentage of code that AI wrote. Google’s 75 percent is undefined, Microsoft’s 30 percent was a verbal estimate, and neither tells you whether anything shipped or held. If you want a number that means something, the CORE framework work on getting better output up front is worth more than instrumenting how much output there was.
The strongest finding in all of this is also the least quotable. The tools work, they work most where people know least, and they shift effort from writing to reviewing. A team that was disciplined about review before will do well. A team that was not is about to find out, 242 percent faster.