Deepfake detection in production

Why deepfake detection benchmarks fall apart on production data

Last year, my team tested a well-regarded deepfake detector against our own production data. The result? The popular model performed only slightly better than a coin flip.

That model was GenD (Yermakov, Cech, Matas, and Fritz, WACV 2026), a detector built on top of the CLIP ViT-L/14 backbone and one of the state-of-the-art public deepfake detectors. Across 14 academic benchmarks it reported an average cross-dataset AUROC of 91.2%, the standard metric of how well a detector separates real images from fakes. On our identity verification data, however, that number dropped to just over 60%.

A result that low is not acceptable in production.

But why does a detector that performs so well on benchmarks do so badly when working with real production data? In this article, I will explain how this gap manifests and detail how Incode replicates a production environment in which to test deepfake detection.

Some context on what production means for us at Incode. Our system does more than look at one image and call it real or fake. Every verification session also checks the user's behavior, the device hardware, and whether the camera feed is genuine, and all of it has to finish in under one second. Building deepfake detection for that environment meant rethinking where our training data comes from, how we label it, and how we keep up with new generators.

Production data is a different domain

Public deepfake datasets and identity verification selfies barely resemble each other, and a detector trained on one struggles with the other. To confirm the GenD result was not a fluke, we ran the experiment in reverse. We took our own production detector, the one we actually rely on, and tested it against more than 10 public deepfake datasets. It correctly rejected fraud, but it also rejected most of the legitimate real images.

The mismatch runs in both directions.

The reason becomes clear once you look at the images side by side. Our production input is a centered selfie, unfiltered, taken straight on, with the face occupying roughly a fifth of the frame. Many public dataset images labeled real are heavily post-processed, cropped, or compressed. A model trained on one kind of image treats the other kind as unfamiliar.

Put simply, identity verification selfies are a narrow slice of all the images on the internet, and public datasets barely cover that slice.

The lesson was simple. Public models and public benchmarks cannot be the basis for a production identity verification system. The research has to happen on our own data: identity verification selfies, captured the way our users actually capture them.

Human-generated labels aren’t reliable at scale

Once you accept that you need to train your own deepfake detector on your own data, the next question is where the training data comes from. Benchmark datasets arrive with clean real-or-fake labels. Production data arrives with none, so we ran an experiment to see whether human labeling could fill that gap.

We asked five labelers who know deepfakes well, including face swaps, fully synthetic faces, and animated faces, to label a set of images. Then we compared their accuracy with our detector's.

The result? Our current model had a lower error rate than the average labeler. A version of our model from two years earlier still beat the labelers, even when we combined their answers by majority vote. Labelers also tended to over-flag real images, often because they read ordinary facial asymmetry as evidence of manipulation.

If a two-year-old model already outperforms expert human labelers, human labeling is not a reliable way to build training data at scale.

So we build our own training datasets instead. They are designed to look like real production images and real fraud, and to include kinds of fraud that have not shown up yet. That means generating known fraud types ourselves, and running internal red-team exercises where ethical hackers probe our own systems for weaknesses.

One experiment stuck with me. We fine-tuned an open-source Stable Diffusion XL model on our internal data to see what it would produce. The output was startlingly realistic: people who looked tired and mildly annoyed, exactly like someone redoing an identity check they have already done once. It was a good reminder that the same generative tools we defend against are also our best way to build realistic training data.

The images on the left were generated by open-source StableDiffusion XL. The images on the right were generated by our fine-tuned model.

Benchmarks only score generators that already exist

A benchmark is a fixed set of fakes made by the generators that existed when it was built. New image generators appear every few weeks, and fraudsters adopt them quickly. In production, we see fakes from a new generator long before any benchmark includes it.

So, in effect, a high benchmark score indicates how well a detector catches last year's fakes. It says nothing about the question that matters: what happens when a generator it has never seen shows up in live traffic?

We cannot retrain from scratch every time one appears, so at Incode we rely on three things to handle new generators:

1. Few-shot adaptation: When a new generator appears, we often only have a handful of samples to work with. Our fix is simple: when the detector misses a fake from a new generator, we teach it to treat anything that looks similar as suspect. That cut our error rate on new generators roughly 10x, without hurting accuracy on the ones we already knew.
2. Automation: We are building an agent-based system that scans the internet for new generative models, evaluates whether they are capable of producing convincing face impersonations, figures out how to access them, and tests our detector against their output automatically. A Slack bot flags new discoveries to the team as they happen, because tracking this manually is no longer realistic.
3. Generalization: The detector catches fakes from tools it was never trained on. Here is the evidence. We looked at how our detector sorts images, and it had split fakes into two clean groups on its own: one for diffusion-based generators and one for GAN-based generators, the two main families of image generation. We never trained it to make that distinction. Then we fed it samples from Nano Banana Pro, a generator that did not exist publicly when the model was trained. They landed almost entirely in the diffusion group. In other words, the detector recognized a tool it had never seen.

Agentic fraud is turning deepfakes into an orchestration problem

Today, most fraud attempts are still manual. Someone generates a deepfake, injects it through a jailbroken device or a virtual camera, sees whether it worked, and adjusts by hand if it does not. That process is slow enough that unusual, repeated attempts tend to get caught.

But incredibly soon, this modus operandi will become outdated. We’re already seeing agentic fraud take precedence over manual fraud. One agent researches the target using information available online. Another tries different prompts and generation models to produce a convincing selfie, judging the output itself. Another tests injection methods, trying different virtual cameras and emulators until one gets through. All of it can run continuously, adjusting to feedback from the verification system in real time.

Within the context of agentic fraud, deepfake detection becomes one piece of a much larger fraud problem, not the whole of it.

That is the shift I think the research community needs to build benchmarks and defenses for over the next few years. It is also why we treat identity verification as a problem that spans the whole capture process, from the device to the camera to the image, rather than a single real-or-fake decision.

Before trusting a deepfake detection score, ask what data it was tested on

If you take one action item from this, make it a question to ask of any deepfake detection number: what data was it measured on? A score from public benchmarks tells you how a model performs on public benchmarks. Until it has been tested in a production environment, on data that looks like your own users, treat it as a starting point and expect it to drop.

I go into more of the technical detail in the full talk, including the embedding-space visualizations and the labeling experiment results, in our most recent webinar. Watch now to learn more about deepfake detection in production.

Ready to learn more about how Incode tests our proprietary models on live production data? Request a demo today.