Every developer now has an opinion about vibe coding, and most of those opinions are calibrated against how the workflow feels. That is a problem, because the best measurement we have of AI-assisted development says feelings and outcomes have come apart: developers in a randomised trial believed AI made them faster while the clock showed the opposite. This article defines the term precisely, then walks through what published studies find about the security, quality and speed of shipping AI-generated code.
What vibe coding is, and is not
The term has a specific origin. In a post on X in February 2025, Andrej Karpathy, a co-founder of OpenAI, described a new style of building software: giving high-level instructions to an LLM-based coding agent, accepting its output largely without reading the diffs, and pasting errors back in for the model to fix. The phrase spread fast enough that Merriam-Webster added it as a slang term within weeks, defining it, with audible eyebrow, as writing computer code in a somewhat careless fashion, with AI assistance. Collins made it a word of the year for 2025.
The definitional core is not “using AI while coding”. Autocomplete-style assistance, where a developer reads and owns every accepted line, is ordinary AI-assisted programming. Vibe coding is the stronger practice: describing what you want and shipping generated code without fully reading or understanding it. This article is about that stronger practice, and about what AI-generated code looks like when it lands in real codebases.
Security: worse code, more confidence
The best-known controlled evidence comes from a Stanford user study (Perry, Srivastava, Kumar and Boneh, presented at ACM CCS 2023). Participants completed security-relevant programming tasks across several languages, half with access to an AI assistant built on OpenAI’s codex-davinci-002 and half without. Those with the assistant wrote significantly less secure code. Worse, they were also more likely to believe their code was secure than the control group.
That second finding is the important one for vibe coding specifically, because vibe coding removes the review step that would catch the problem. The model underlying the study is dated now, and newer models write better code; but the psychological result (assistance inflates confidence in security beyond what the code merits) is a property of the workflow, not the model version, and no published study yet shows it has gone away. If generated code ships unread, the security review has effectively been delegated to a system that the study’s participants trusted more than it deserved.
Speed: the perception gap, measured
In 2025, METR ran the study everyone should have run earlier: a randomised controlled trial of experienced developers doing real work. Sixteen developers with moderate AI experience completed 246 tasks in mature open-source projects they knew well, averaging 5 years of prior experience on those repositories. Each task was randomly assigned to allow or disallow AI tools (mostly Cursor Pro with Claude 3.5/3.7 Sonnet).
Before starting, developers forecast AI would cut their completion time by 24%. Afterwards, they estimated it had cut it by 20%. The measured result: allowing AI increased completion time by 19%.
The caveats matter and cut both ways. This was one setting: experts, on codebases they knew intimately, where the model’s context disadvantage against a five-year veteran is at its maximum. It says little about greenfield prototypes, unfamiliar languages or novice programmers, where speedups are plausible. But it demolishes the idea that perceived productivity is a trustworthy measurement. A 39-percentage-point gap between belief (20% faster) and reality (19% slower), in the population best placed to judge, means “it feels faster” should carry roughly zero evidential weight in your team’s tooling decisions.
Quality and churn: what lands in repositories
For what happens after code ships, the largest dataset comes from GitClear, which analyses structured code-change data across commercial and open-source repositories. Its analysis of 211 million changed lines from 2020 to 2024, spanning the arrival of AI assistants, found that lines associated with refactoring fell from 25% of changed lines in 2021 to under 10% in 2024, while copy/pasted (cloned) lines rose from 8.3% to 12.3%. Its earlier report, on 153 million changed lines through 2023, projected that code churn, meaning lines reverted or updated within two weeks of being written, would double in 2024 versus the 2021 pre-AI baseline. Independent reporting on the 2025 edition highlighted an eightfold increase during 2024 in code blocks containing five or more duplicated lines.
GitClear is a vendor of code-analysis tools, so treat the framing with that in mind; but the underlying pattern (more code added, more duplication, less consolidation and reuse) is exactly what you would predict when generation is cheap and review is optional, and it is the pattern maintainers report. These are correlational, ecosystem-level signals, not a randomised trial; nobody has yet run a long-term controlled study of maintenance cost for vibe-coded versus reviewed code.
Where that leaves the honest conclusion
Three measured findings, one theme. Security: AI-assisted participants wrote less secure code while feeling more secure. Speed: experienced developers were slower with AI while feeling faster. Codebase health: churn and duplication rose across the industry while everyone felt more productive. The consistent gap is between subjective experience and measured outcome, and vibe coding by definition strips out the practices (reading, review, testing you wrote yourself) that would close it.
That does not make the practice worthless. For throwaway prototypes, personal tools and low-stakes experiments, careless-by-design is a fair trade, which is roughly the spirit of Karpathy’s original post. The evidence problem starts when unread code enters products with users, security surfaces and a maintenance future.
Common questions
What is vibe coding?
Vibe coding means building software by describing what you want to an LLM and largely accepting the generated code without reading or fully understanding it. Andrej Karpathy coined the term in February 2025, and Merriam-Webster lists it as slang for writing code in a somewhat careless fashion with AI assistance.
Is vibe coding the same as using Copilot or autocomplete?
No. With autocomplete-style assistance the developer reads and takes ownership of every accepted line. Vibe coding specifically means shipping generated code without full review, and the research on insecure output and rework applies most strongly to that unreviewed mode.
Does AI actually make developers faster?
Not always, and perception is unreliable. In METR’s 2025 randomised trial, 16 experienced developers doing 246 real tasks were 19% slower with AI tools while believing they were 20% faster. Speedups remain plausible for unfamiliar code or prototyping, but that has not been demonstrated with comparable rigour.
Is AI-generated code less secure?
In the Stanford user study, participants with an AI assistant wrote significantly less secure code on security-relevant tasks and were more confident it was secure. Newer models perform better than the one studied, but no published trial shows the overconfidence effect has disappeared, so unreviewed generated code should be treated as untrusted input.
The bottom line
Use generation freely for prototypes and code you will throw away. For anything that ships, the data support one rule: AI can write the code, but a human still has to read it. And when someone on your team says the tools obviously make everyone faster, remember that the developers in the only randomised trial to date said the same thing, and were 39 percentage points wrong.
Did this answer your question?
Thanks. That helps us decide what to fix.
Leave a comment