Deepfake detectors that read a pulse signal learn the video source

Researchers tested deepfake detectors that read a heartbeat in face video, with remote photoplethysmography (rPPG), which finds pulse in small color changes of skin. On talking face fakes, simple color traces got an AUC (a score where 1.0 is perfect) of 0.998, and a dedicated pulse tool got 0.87. The tests show that the signal is not a pulse.

How it works

The signal is as strong in background regions as on the face. It is above the heart rate band. Remove high frequency content, and most fake videos look not fake.

The numbers

Detectors that use Celeb-DF++ as training data classify more than 99% of FF++ videos that are not fake as fake. The official Celeb-DF splits share identities and source video between training and test. Custom protocols that use test video again in training make weaker detectors look better by a maximum of 0.30 AUC.

What is not known

The study uses two data sets and one sampled window for each video. A frozen CLIP backbone, a general image model, equals or beats mixes of CLIP with pulse features. A different reimplementation got the same results.

Insight: High detector scores can show a data match, not a fake find

Earlier reports give 97.04% for PairCLIP-SWA and 94.8% for MoDA. MoDA got 78.13% on 1,200 unseen Internet videos. This paper gives a cause: detectors learn the source of a video. A high score seems to show that test data and training data agree. Buyers who tell vendors to test on a new source get correct numbers.