Anthropic and METR research shows risk to training new engineers from heavy AI use

The promise that didn't hold

About a year before this talk at QCon London, at a conference in New York, Dario Amodei, Anthropic co-founder, said that within three to six months AI would write 90% of all code produced, and that within 12 months it would be virtually 100%. Six months in, he himself declared the prediction confirmed, citing the 90% figure inside Anthropic. This is reported in Alasdair Allan's talk at QCon London, published by InfoQ.

A year after the original prediction, an analysis by Redwood Research concluded that the 90% figure was misleading: counting only code actually committed to repositories, the share is close to 50%. It only reaches 90% when disposable scripts and explorations that never made it into production are added in. Allan compares this with internal numbers from other companies: Google reports just over 25% of internal code generated by AI, Microsoft is close to 30%, and GitHub Copilot has a suggestion acceptance rate of around 30% in corporate environments. None of these numbers changed meaningfully in the six months before the talk.

In short: AI did change the volume of code written, but not in the essentially total way Amodei predicted. The question that interests Allan isn't how much code AI writes, but what happens to engineer training when the step of writing code, historically how engineering was learned, starts being done by a machine.

The supervision paradox

In July 2025, METR (Model Evaluation and Threat Research) published a randomized controlled study, not vendor research, that produced the Reuters headline "AI slows down some experienced software developers, study finds." Experienced developers using AI tools took 19% longer to complete tasks than they would have without AI. Before starting, they predicted AI would make them 24% faster; after finishing, they still believed they had been 20% faster. The gap between perception and reality reached 43 percentage points.

The follow-up is more revealing than the original result. In February 2026, METR tried to repeat the study and couldn't: not because of a methodology failure, but because it couldn't find developers willing to work half the tasks without AI. A growing share of participants refused to take part in any study that required that. In under a year, dependence stopped being optional and became structural before anyone truly understood what the tool was doing to those who use it.

I built this now means I prompted this.

Alasdair Allan, CTO of Negroni Venture Studios, in a talk at QCon London

What Anthropic's study with juniors showed

Anthropic ran its own controlled trial: 52 engineers, most of them juniors, learning the Python Trio library, with or without AI assistance. Completion time showed no statistically relevant difference between the two groups (p-value of 0.391): AI did not measurably speed up learning.

What showed a statistically significant difference (p-value of 0.01) was the result of a domain test applied afterward. The group that used AI scored 50% correct versus 67% for the group without AI, a 17-point drop, concentrated precisely on debugging questions, the skill needed to verify whether AI-generated code is correct.

The productivity gain from AI assistants on this task failed to reach statistical significance.

Excerpt from the Anthropic paper on the Trio library experiment, cited in Alasdair Allan's talk

Anthropic itself named the phenomenon the "supervision paradox": using AI requires human supervision, and supervision requires exactly the code-reading and debugging skills that atrophy from excessive use of AI itself. Allan identified three failure patterns among those who scored low on the test:

• Total delegation: didn't even try to solve it alone, relied on AI from the start.
• Progressive dependence: tried for a while and then came to depend entirely on AI.
• Iterative AI-on-AI debugging: used AI to fix the very code AI had generated, without understanding what had been done.

All engineers in these three groups scored below 40% correct, versus 67% for the group without AI.

Where the progression path breaks

METR also measures what it calls the "50%-reliability task horizon": the length of a human task that an AI agent can complete on its own, with a 50% chance of success. This horizon has doubled every seven months since 2019, and every four months since 2024. But the success curve drops sharply after about an hour of task length: AI gets right nearly 100% of tasks lasting a few minutes, typical of a junior's first year, and less than 10% of tasks lasting more than four hours, which depend on accumulated judgment.

Another measurement from METR itself shows why this number overstates real capability: across 18 tasks from mature open source projects, Claude 3.7 Sonnet produced no acceptable pull request in 38% of cases, even when passing automated tests. Every generated PR had at least three quality issues (documentation, test coverage, or lint), and fixing a PR until it became acceptable took an average of 42 minutes, about a third of the time a human would spend doing the task from scratch.

Two market surveys cited by Allan show the same bottleneck shifting location:

The bottleneck doesn't disappear: it moves up to code review, exactly where senior judgment lives. And it's that judgment that massive adoption among juniors isn't building.

The Amazon Kiro case

Allan uses a real incident to illustrate the risk. In mid-December 2025, AWS engineers gave Kiro, Amazon's AI coding tool, operator permissions equivalent to those of a human engineer to modify AWS Cost Explorer. The agent decided the best approach was to delete and rebuild the entire environment instead of making targeted changes, and AWS took 13 hours to restore the service in one of its China regions.

Amazon attributed the incident to an access-control configuration error. Four sources later interviewed by the Financial Times, according to the talk, said it was Kiro itself that made the decision to delete and rebuild the production environment autonomously, something an experienced human engineer would be unlikely to do, precisely because they had already seen that kind of operation go wrong before.

What this means for those hiring and training devs in Brazil

Allan's framing serves as a direct warning for Brazilian squads that today put juniors fresh out of bootcamp to work mostly with coding agents, without a structured senior review routine. If AI's success curve drops sharply after an hour of task length, it's exactly on long, ambiguous tasks, the ones that build judgment, that Brazilian juniors need to keep being assigned to code without assistance, even if that looks slower in the short term.

For engineering leaders, the Faros AI data on PRs 154% larger and reviews 91% slower is a practical warning: the bottleneck moved to code review, and an overloaded senior reviewer tends to approve large PRs out of fatigue, not confidence. None of the studies cited by Allan recommends banning AI; the point is that using the tool without redesigning how juniors practice debugging and legacy code reading tends to produce, within a few years, a team with no one capable of supervising what AI itself produces.