If people cannot reliably spot deepfakes, the obvious answer is to let software do it. A small industry now sells exactly that: upload a video, a voice clip or an image, and get a verdict. The marketing quotes accuracy figures in the high nineties.
Those figures are usually true, and usually irrelevant. They describe performance on tidy academic datasets, which bear about as much resemblance to a fake circulating on social media as a driving test does to rush hour. In 2025 a group of researchers tested detectors on the real thing. The results are the best available guide to what these tools can and cannot do for you.
Why lab scores mislead
A detector is a model trained to tell real media from fake. It learns from examples, and the examples come from published datasets: thousands of videos made with a handful of known face-swap techniques, in good resolution, with clean labels.
Tested on more of the same, such a model does brilliantly. The trouble is what it has actually learned. It has learned the fingerprints of those particular generators. A fake made with a newer tool, then compressed by a messaging app, cropped, re-encoded and screenshotted, carries different fingerprints or none.
The authors of the 2025 study put it directly: while many deepfake detectors report high accuracy on academic datasets, these academic benchmarks are out of date and not representative of real-world deepfakes.
The real-world test
The study came from TrueMedia.org, working with researchers at the University of Washington. They assembled a benchmark called Deepfake-Eval-2024 from material that people had actually encountered and submitted: in-the-wild deepfakes collected from social media and deepfake detection platform users in 2024.
It is large and messy in the way reality is. It consists of 45 hours of videos, 56.5 hours of audio, and 1,975 images, drawn from 88 different websites in 52 different languages. The fakes include lip-syncs, face-swaps and fully generated images.
They then ran leading open-source detectors against it. Performance, in the authors’ words, drops precipitously, with AUC decreasing by 50% for video, 48% for audio, and 45% for image models compared to previous benchmarks.
AUC is a standard score for classifiers where 1.0 is perfect and 0.5 is what you get from guessing. A detector that scores near-perfect in the lab and then loses half of that is not slightly worse. It is approaching the point where its output tells you almost nothing.
Commercial tools do better, within limits
The study also tested commercial detectors and open-source models retrained on the new data. Both did better than the off-the-shelf models. A follow-up paper summarises the figures: three finetuned open-source models achieve accuracies between 61% and 69%, significantly lagging behind the leading commercial deepfake detector with 82% accuracy.
Eighty-two percent is a real improvement and a long way from the ninety-nine on the sales page. It means roughly one wrong verdict in every five or six.
Then comes the line that should shape how you use these tools. The commercial and fine-tuned models, the authors report, do not yet reach the accuracy of deepfake forensic analysts. Trained human specialists, working slowly with context, provenance checks and multiple tools, still beat the best software.
That is not a contradiction of the finding that ordinary people detect fakes at near-chance rates. An untrained person glancing at a clip and a forensic analyst spending an hour on it are doing different jobs. It does mean the realistic role of detection software is as one instrument in an expert’s kit, not an oracle for the rest of us.
The structural problem
Three features of the problem suggest this will not simply be fixed by the next release.
It is an arms race with a built-in lag. A detector is trained on the output of existing generators. A new generator arrives, and the detector is behind until someone collects examples of the new fakes, labels them and retrains. The attacker moves first by definition.
Detectors can be used as training signals. Anyone building a generator can test its output against public detectors and adjust until it passes.
Ordinary handling destroys evidence. The subtle statistical traces detectors rely on are degraded by compression, resizing and re-recording, which is what happens to every file as it spreads.
There is a hopeful note in the research. In one face study, the visual attributes that fooled human viewers still permitted high accuracy using machine learning. Machines can see regularities that people misread. The capability is real. It is the generalisation to new, degraded, real-world fakes that fails.
What a detector verdict is worth
Think about it the way you would think about any imperfect test.
A “fake” verdict on something you already had reason to doubt adds weight. A “real” verdict on something surprising proves little, because the failure that matters most (a new kind of fake the tool has never seen) is precisely the one it cannot warn you about.
False alarms carry their own cost. A tool that is right 82% of the time will sometimes flag genuine footage. As detection tools spread, “the software says it’s fake” becomes a convenient way to dismiss real evidence, a problem serious enough to have its own name, the liar’s dividend.
What is not known
How current commercial tools perform. The benchmark covers 2024 fakes. Both generators and detectors have moved since, and independent, up-to-date, real-world testing is scarce. Vendors rarely submit to it.
How tools perform on the specific content you care about. Performance varies by type (video, audio, image), by language and by how the file has been handled. A headline accuracy figure averages over all of that.
Whether real-time detection works. Catching a live deepfake on a video call is a harder problem than analysing a file, and public evidence on it is thin.
How to use these tools sensibly
Treat the verdict as one input. Combine it with source, context and corroboration. Never let it be the only thing between you and a decision about money, access or someone’s reputation.
Ask vendors the right question. Not “what is your accuracy?” but “what is your accuracy on in-the-wild media you did not train on, and who measured it?”.
Use more than one. Different detectors fail differently. Agreement between independent tools means more than a single confident score, and disagreement is a signal to slow down.
For anything that matters, escalate to a person. News organisations and fraud teams use human analysts for a reason the data supports.
Build defences that do not depend on detection. Call-back procedures, two-person approval and verified channels work against a perfect fake. A detector, by construction, does not.
Common questions
Can software detect deepfakes?
Sometimes. On a benchmark of real 2024 deepfakes, leading open-source detectors saw their AUC fall by 50% for video, 48% for audio and 45% for images compared with academic benchmarks. The best commercial detector reached 82% accuracy.
Why are deepfake detectors less accurate in real life?
They are trained on academic datasets that the researchers describe as out of date and not representative of real-world deepfakes. Real fakes use newer techniques and are compressed and re-encoded as they spread, which removes the traces detectors look for.
Are paid deepfake detectors better than free ones?
In the 2025 benchmark, commercial models and models retrained on real-world data outperformed off-the-shelf open-source models: 82% for the leading commercial tool against 61% to 69% for fine-tuned open-source ones. None matched human forensic analysts.
Should I trust a deepfake detector’s result?
Only as one piece of evidence. At 82% accuracy roughly one verdict in five or six is wrong, and a tool is weakest against new kinds of fake it was never trained on.
The bottom line
Deepfake detection software works well on the fakes it has seen and poorly on the ones it has not, and the ones you will meet are the second kind. Tested on real material from 2024, open-source tools lost about half their accuracy and the best commercial system was wrong about one time in five. Use detectors as a second opinion, never as a verdict, and put your real protection in procedures that still hold when the fake is perfect.
Did this answer your question?
Thanks. That helps us decide what to fix.
Leave a comment