Paste a paragraph into an AI detector and it hands back something that looks like a measurement: “87% likely AI-generated”. It has a decimal point. It has a colour. It feels like a lab result.
It is not one. The number is a guess from a statistical model about how predictable your wording is, and the independent studies that have tested these tools are close to unanimous. They miss most AI text when it has been lightly edited, and they accuse real people, some groups far more than others. If you manage a team, mark essays, review applications or are simply tempted to “check” a colleague’s document, this is what the evidence says about the tool you are about to trust.
The company that built the most famous model gave up
In January 2023, OpenAI released its own classifier for spotting AI-written text. It was candid about the performance. In its own evaluations the tool correctly identified 26% of AI-written text as “likely AI-written”, while incorrectly labelling human-written text as AI-written 9% of the time.
Read that again: it caught about a quarter of the AI text, and wrongly flagged roughly one human writer in eleven. Six months later the company withdrew it. The notice read: as of July 20, 2023, the AI classifier is no longer available due to its low rate of accuracy.
If the organisation with the most knowledge of how the text is generated could not build a reliable detector, that should set expectations for everyone else.
What independent testing found
The most thorough independent test was published in the International Journal for Educational Integrity in 2023. Debora Weber-Wulff and seven co-authors tested 14 AI-text detectors, 12 public tools plus the commercial systems Turnitin and PlagiarismCheck.
Every tool scored below 80% accuracy and only five crossed 70%. Six of 14 tools produced false positives and 13 of 14 produced false negatives. Accuracy degraded further on paraphrased and machine-translated text, which is to say on exactly what someone trying to hide AI use would submit.
The authors’ verdict was unusually blunt for an academic paper: the available detection tools are neither accurate nor reliable.
Note the direction of the main error. The tools were biased towards calling AI text human. So a detector fails the person relying on it in both directions: it lets through the people who made a small effort to disguise their work, and it still catches some who did nothing wrong.
Who gets falsely accused
The false positives do not fall evenly. A Stanford team tested seven widely used detectors on two sets of genuine human writing: essays by US eighth-graders, and 91 essays written by non-native English speakers for the TOEFL English proficiency exam.
The detectors were near-perfect on the American students. On the TOEFL essays, they misclassified over half as “AI-generated”, with an average false positive rate of 61.22%. All seven detectors unanimously identified 18 of the 91 TOEFL essays (19.78%) as AI-authored, and 89 of the 91 TOEFL essays (97.80%) were flagged by at least one detector.
Every one of those essays was written by a person.
The explanation lies in how the detectors work. Most lean on perplexity, a measure of how predictable each word is given the ones before it. Language models produce low-perplexity text because they choose likely words. People writing in a second language also tend to use common words and simple constructions. The detector cannot tell a limited vocabulary from a language model.
The authors’ practical conclusion was to be extremely careful about using these detectors, given the stakes when they are applied to job applications, college entrance essays or high school assignments.
The same paper showed how fragile the tools are in the other direction: simple prompting strategies could bypass GPT detectors.
A fair caveat: this finding may not be universal. A 2026 study that repeated the exercise with non-native speakers of Czech reported that its findings differ significantly from those in the prior study. The bias is well documented in English with the 2023 generation of detectors. How far it extends is still being worked out.
Why a small error rate is still a big problem
Vendors respond that their tools are better than the ones in these studies. Turnitin, for example, claims a false positive rate of less than 1%.
Suppose that is true. The arithmetic of a rare accusation applied to a large group is unforgiving. A university that runs 20,000 genuine, human-written submissions through a detector with a 1% false positive rate should expect about 200 students to be flagged for something they did not do. Each of those is an honest person facing a misconduct process with no way of proving a negative. You cannot show the drafts that prove you did not use a tool.
This is the same base-rate problem that affects medical screening. When the thing you are testing for is uncommon and the consequences of a false alarm are serious, even a good test does more harm than its accuracy figure suggests.
What is not known
How current commercial detectors perform under independent test. The published studies used 2023 tools and 2023 language models. Both have moved. Vendors publish their own figures, but independent replication on current systems is thin, and the structural problem (detectors train on yesterday’s models) has not gone away.
Whether any text watermark will be widely adopted. A watermark built into the generator is a sounder approach than guessing afterwards, but it only works if the model that wrote the text applied one and nobody paraphrased it since.
Whether the bias against non-native writers persists in newer tools and other languages. The Czech result suggests it may vary.
What to do instead
Never act on a score alone. The evidence does not support treating a detector percentage as proof of anything. At most it is a reason to have a conversation.
Assess the person, not the text. If you need to know whether someone understands what they submitted, ask them to explain it, extend it, or defend a decision in it. Five minutes of questions tells you more than any classifier, and it works whether or not AI was involved.
Look at process where you can. Version history, commit logs, drafts and notes are evidence of how something was made. They are imperfect, but they are evidence about the actual work and not about word statistics.
Set expectations up front. Much of the conflict comes from unclear rules. If AI assistance is allowed for some things and not others, say so, and say what disclosure you expect.
If you are accused, ask what the tool’s false positive rate is, who validated it, and whether it has been tested on writers like you. Offer your drafts and offer to talk through the work. The studies above are citable, and the burden should sit with the accuser.
Common questions
Are AI detectors accurate?
Not reliably. An independent test of 14 detectors found every tool scored below 80% accuracy and only five exceeded 70%, with 13 of 14 producing false negatives and six producing false positives. The authors concluded the tools were neither accurate nor reliable.
Can AI detectors falsely flag human writing?
Yes. In one study, seven detectors misclassified TOEFL essays by non-native English speakers as AI-generated at an average rate of 61.22%, and 97.80% of those human-written essays were flagged by at least one detector.
Why did OpenAI shut down its AI detector?
OpenAI withdrew its classifier in July 2023 citing its low rate of accuracy. By the company’s own figures it correctly identified 26% of AI-written text and wrongly labelled human text as AI-written 9% of the time.
Can you get around an AI detector?
Easily, according to the research. Detector accuracy fell on paraphrased and machine-translated text, and the Stanford study found simple prompting strategies were enough to bypass detection.
The bottom line
An AI text detector gives you a confident number built on a weak signal. It misses disguised AI text, it flags honest writers, and in the best-known study it flagged non-native English speakers most of all. Use it, if at all, as a prompt for a conversation and never as a verdict. If you need to know whether someone did the work, ask them about the work.
Did this answer your question?
Thanks. That helps us decide what to fix.
Leave a comment