Deepfake

How it works

MITRE ATLAS files Generate Deepfakes under AI Attack Adaptation, the tactic for turning general capabilities into attack-ready outputs. The attacker obtains a generative tool, gathers photos, video or recordings of the person to be faked, and generates media that passes as them. ATLAS notes that the tools may be open-source software built for legitimate use, though some, such as ProKYC, are made for crime.

The output is then aimed one of two ways. Aimed at a person, it becomes phishing: a cloned voice on a phone call, or a faked executive on a video call. ATLAS says a voice can be cloned from a few seconds of public audio. Aimed at a model, it becomes model evasion: a face-swapped video fed to a selfie check in place of the real camera.

Why it matters

Faking media is old. ATLAS’s point is that generative AI reduces the skill and effort needed, so attackers can target more people and systems, and can now manipulate media in real time.

Detection helps but does not settle the problem. ATLAS lists deepfake detection as a mitigation: models trained to tell real from fake, checks for flaws such as unnatural facial movement or mismatched audio, and biometric cues such as blinking. NIST calls automated detection “a constant cat-and-mouse game”: generators improve as soon as a new detector appears, and detectors are often tied to specific generators and may work well only on those. ATLAS also lists extra sensors, such as infrared depth cameras, as an aid.

In practice

Two ATLAS case studies show the biometric route. In an exercise, the iProov red team fed a live face-swapped video into a phone in place of its camera. It got past facial recognition and the liveness checks meant to confirm a real person is in front of the camera. In an incident reported by Cato CTRL, a tool called ProKYC was being sold to criminals to make fake identity documents with matching deepfake selfie videos, to open accounts on services such as cryptocurrency exchanges.

Where definitions disagree

ATLAS and the EU AI Act draw the boundary in different places. ATLAS’s Generate Deepfakes technique covers “imagery, video, audio, and text”, mimicking a real person or depicting a fictional one. Even within ATLAS the wording varies: its deepfake-assisted phishing entry describes deepfakes as “AI-generated synthetic images, audio, or video”.

The EU AI Act defines a “deep fake” for disclosure duties rather than threat modelling, and more narrowly: “AI-generated or manipulated image, audio or video content that resembles existing persons, objects, places, entities or events and would falsely appear to a person to be authentic or truthful”. Text is not in that list, and the test turns on resemblance to something that exists. This page follows the Generate Deepfakes technique, because it describes deepfakes as a security threat.

Questions and answers

Is a deepfake an attack on an AI system?

Sometimes. MITRE ATLAS lists two uses for a generated deepfake. One targets people: impersonating an executive or colleague in a phishing call or message. The other targets a model: presenting a faked face or voice to a biometric identity check so the system accepts it as genuine, which ATLAS counts as evading an AI model.

Can deepfake detection tools be relied on?

Not on their own. NIST describes automated detection as a constant cat-and-mouse game in which generators improve as soon as a new detector appears, and notes that detectors are often tied to specific generators and may perform well only on those. NIST also warns that in many contexts flagging real content as AI-generated can be extremely damaging, so a detector's false positives matter, not only what it misses.

Does the EU AI Act definition of a deep fake include text?

No. Article 3(60) of the EU AI Act limits a "deep fake" to image, audio or video content that resembles existing persons, objects, places, entities or events. MITRE ATLAS's Generate Deepfakes technique is wider and lists text alongside imagery, video and audio.