All articles

Journal

Human + LLM > human alone > LLM alone: why mastering your software is still the best guarantee

The studies agree. A developer working with an LLM produces more than a developer alone, who produces better than an LLM alone. But the ordering holds on one condition, and it is the one that matters to a client.

Date
Reading
10 min read
Topics
aillmengineeringqualitysecurity

There are two easy stories about language models and software. The first says the developer is on borrowed time: the model writes the code, the human watches. The second says it is all a toy that produces brittle code and that a real engineer does without it.

The evidence available in 2026 supports neither. It draws a fairly clear ordering:

human + LLM > human alone > LLM alone

… with a clause both stories leave out: the ordering only holds when the human masters the software being produced. Remove the mastery, and the pair drops below the human alone. That is the condition that matters, and it is precisely what a client is buying when they hand a system to a contractor.

LLM alone: fast, plausible, and nobody to answer for it

Start at the bottom. "LLM alone" does not really exist in production: someone always presses merge. But vibe coding, where the model's output is accepted without being read, is an honest approximation. What do we know about that code?

  • Veracode's 2025 GenAI Code Security Report tested more than 100 models on 80 tasks across Java, JavaScript, Python and C#. 45% of samples introduced an OWASP Top 10 vulnerability. In Java, 72%. The most uncomfortable finding: newer and larger models did no better than smaller ones; security is not a problem that gets fixed by waiting for the next release.
  • Stanford's Do Users Write More Insecure Code with AI Assistants? (Perry et al., CCS 2023) found that participants with an assistant wrote less secure code on four of five tasks… and were more convinced they had written secure code than the control group. The flaw and the confidence in the flaw arrive together.
  • On maintainability, GitClear's 2025 analysis of 211 million changed lines between 2020 and 2024 measured an eightfold rise in duplicated blocks and a collapse of moved or refactored code, from roughly 25% of changed lines to under 10%. 2024 is the first year in the dataset where copy-paste outweighs refactoring.

None of these numbers say the model is useless. They say that code produced without competent review is code whose security and evolution nobody can guarantee, and that when an incident hits, there is nobody who understands the system well enough to fix it. Judgement is exercised in two places: in review, on the diff, and later, on what you can still understand five years on.

Human + LLM > human alone: the gain is real

The top of the ordering is better documented than people assume, and by controlled trials, not surveys.

  • The GitHub Copilot RCT (Peng et al., 2023): 95 developers, an HTTP server to implement in JavaScript. The assisted group finished 55.8% faster.
  • Google's internal RCT (Paradis et al., 2024): 96 engineers, a realistic C++ task with completion, smart paste and chat. Estimated time saving of about 21%, larger for those who spend the most hours coding.
  • The BCG / Harvard field experiment Navigating the Jagged Technological Frontier (Dell'Acqua et al.): 758 consultants. On tasks within the model's reach, assisted consultants completed 12% more tasks, 25% faster, with quality rated more than 30% higher. The 2023 working paper put it at 40%; the version published in Organization Science in 2026 brought it down to more than 30%, and that is the one we cite.
  • The meta-analysis by Vaccaro, Almaatouq and Malone in Nature Human Behaviour (2024), across 106 experiments and 370 effect sizes: compared with humans alone, human-AI pairs win with a medium-to-large effect (Hedges' g = 0.64).

On this point the debate is over: an equipped developer produces more than one who is not.

… provided the human is better than the machine at the task

This is where the same meta-analysis gets interesting, and uncomfortable. Vaccaro and colleagues do not only compare the pair with the human alone; they compare it with the better of the two on their own. There, on average, the pair loses (g = −0.23). The breakdown is worth reading:

  • when the human alone is better than the AI alone at the task, the combination gains (g = 0.46);
  • when the AI alone is better than the human alone, the combination loses (g = −0.54).

In other words, the pair's added value comes from a human who knows when not to follow the machine. A human who cannot judge the output adds nothing: they subtract.

The BCG study tells the same story from another angle. On a task deliberately chosen outside the model's reach, assisted consultants were 19 percentage points less likely to produce a correct answer than those working alone. The model makes you faster and wrong at the same time as soon as you leave its frontier, and the frontier is jagged; you cannot see it from inside.

Two more results belong side by side:

  • METR's controlled trial (July 2025): 16 experienced open-source maintainers, 246 tasks in their own repositories, five years of tenure on average. With early-2025 tools they were 19% slower, while estimating they had been 20% faster. The February 2026 update announced a change of protocol: the second wave, ten developers from the first plus forty-seven new ones, estimates an 18% gain for the former and 4% for the latter, with confidence intervals that contain zero. METR treats those figures as a lower bound, because the most convinced developers now decline to take part in a study that requires working without AI. The 2025 trial remains the last clean randomised measurement available, and its replacement does not exist yet.
  • Microsoft Research's survey The Impact of Generative AI on Critical Thinking (Lee et al., CHI 2025), of 319 knowledge workers: the higher the confidence in the tool, the less critical thinking the person applies; the higher the confidence in themselves, the more.

The conclusion fits in one sentence: the pair is worth what the human's judgement is worth. Not their enthusiasm, not their seniority, their judgement on this specific code.

The trap: mastery erodes when you stop exercising it

If the guarantee rests on human judgement, the question becomes: what does heavy use of the model do to that judgement?

Anthropic (who have no interest in painting a dark picture) published in February 2026 a controlled trial on skill formation: 52 developers learn a Python library they do not know, half with an assistant, half with documentation and search only. On a quiz about the concepts they had just used, the assisted group scored 17% lower. But not all usage is equal: those who used the assistant to understand (asking why this pattern, what this line does) retained as much as the control group. Those who delegated forgot.

Nothing new under the sun. Lisanne Bainbridge described the control-room paradox in 1983 in Ironies of Automation: the more you automate, the more the operator merely monitors, the more their manual skills decay, and those are exactly the skills demanded of them on the day the automation fails. An experienced operator who no longer practises becomes an inexperienced one.

Developers feel it. In the 2025 Stack Overflow survey, 84% use or plan to use AI tools, yet 46% distrust their accuracy (up from 31% a year earlier), and the top frustration, cited by 66%, is output that is "almost right, but not quite". 45% say debugging generated code takes longer than writing it themselves. "Almost right" is the worst case: it passes the obvious tests and breaks where nobody is looking.

Kasparov already put it into words

In The Chess Master and the Computer (2010), Garry Kasparov recounts a 2005 freestyle tournament in which humans and machines could team up freely. Teams of grandmasters backed by several engines dominated the engines alone. But the tournament was won by two American amateurs and three computers, whose strength was knowing how to make their machines work. His formula:

"Weak human + machine + better process was superior to a strong computer alone and, more remarkably, superior to a strong human + machine + inferior process."

Translated into software engineering: what gives the pair its value is neither the model nor the CV, it is the process, and a process is what makes a result reproducible and verifiable by someone else.

At organisation scale, it shows up in delivery metrics

Individual studies tell one story; organisations tell another that points the same way.

Google's 2024 DORA report measured that a 25% increase in AI adoption was associated with a 1.5% drop in delivery throughput and a 7.2% drop in stability, even as developer satisfaction and perceived productivity rose. The authors' hypothesis: producing code faster inflates batch size, and large batches break more.

The 2025 DORA report sums up the sequel in one word: AI is an amplifier. Teams that already have solid foundations (small batches, tests, review, an internal platform) accelerate. Teams carrying debt and vague processes watch their problems grow: more change failures, more rework. The time saved writing is redeployed to verifying, and if nobody is able to verify, it is lost.

What this means when you hand a system to someone

Bring all of this back to the client. When you buy software, you are not buying lines; you are buying someone's ability to answer for those lines in two years, at three in the morning, when the vulnerability is public or the dependency abandoned. The studies above say where that ability lives:

  1. In the human who understands the system. That is the Vaccaro condition: the pair wins when the human is better than the machine at the task. A contractor who cannot explain every architectural decision in their code cannot guarantee it either.
  2. In a process that verifies rather than trusts. Systematic human review, tests written before or alongside the code, small batches, reproducible environments. That is Kasparov's lesson and DORA's: process beats raw strength.
  3. In a practice that keeps the skill alive. Anthropic's study shows that usage aimed at understanding preserves mastery, and usage that delegates erodes it. Bainbridge said it forty years ago. A team that never writes code by hand any more silently loses the ability to judge the code it is offered.

At adnap we use these tools every day: it would be absurd to forgo the speed-up we observe on the tasks where it is real, and we do not put a figure on it here because none of the studies above measures that scope. But we ship nothing we cannot re-read, explain and fix without the model. The LLM speeds up the writing; it does not carry the responsibility. That responsibility is human mastery of the software, and it is the only guarantee we know how to give honestly.

Sources

Ready to take your technology back?

Tell us where you are stuck. A real person reads every message and answers within 48 working hours, with a concrete first step.

Talk to us