Skip to content

Unleashing Vision-Language Semantics for Deepfake Video Detection

🕒 Published (v1): 2026-03-25 16:05 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; VLM semantic priors boost generalizable deepfake video detection

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLAForge is a deepfake video detection (DFD) framework built on CLIP that moves beyond visual-only adaptation to exploit the intrinsic cross-modal (vision–language) semantics of VLMs. It introduces two components — ForgePerceiver and Identity-Aware VLA Scoring — that jointly capture granular forgery cues and identity-conditioned patch-level authenticity signals. On nine cross-dataset benchmarks covering both face-swapping and full-face generation forgeries, it surpasses all state-of-the-art methods.

Problem

Existing CLIP-based DFD methods adapt only the visual encoder (via adapters, bias mitigation, or temporal decoders), leaving the rich textual–visual alignment latent space of the VLM unexploited. Vanilla CLIP's attention distributes over task-irrelevant semantic objects rather than forgery artifacts, and prior VLA-based attempts (RepDFD, FFTG) use only global (image-level) cosine similarity, missing fine-grained patch-level authenticity correspondence.

Method

VLAForge wraps a frozen CLIP backbone with two modules:

ForgePerceiver — a lightweight, independent ViT that receives CLIP patch embeddings \(V \in \mathbb{R}^{h_v \times w_v \times d_v}\) and \(q\) learnable query tokens \(Q \in \mathbb{R}^{q \times d_v}\). It produces: 1. Forgery-aware masks \(M_i = \hat{Q}\hat{V}_i^\top\) (\(i=1,\ldots,H\)) averaged and orthogonality-constrained (\(\mathcal{L}_\text{orth}\)) to encourage diverse artifact priors. These masks are injected as attention biases into CLIP's self-attention layers, augmenting class-token representations for global-level scoring \(s_g\). 2. Forgery localization map \(M_\text{loc} = h([\tilde{M}_1,\ldots,\tilde{M}_q])\) supervised against ground-truth pixel masks via MSE loss \(\mathcal{L}_\text{loc}\).

Identity-Aware VLA Scoring — constructs prompt pairs "This is a real/fake photo of <id> person." where the <id> placeholder token is replaced at the embedding level by the CLIP class-token embedding \(z^{(L)}\) of the query frame, yielding ID-aware text features \(F_r, F_f \in \mathbb{R}^{d_t}\). A VLA attention map is computed as: $\(M_\text{VLA}(i,j) = \frac{\exp(\varphi(P(i,j))F_f^\top)}{\sum_{c\in\{r,f\}}\exp(\varphi(P(i,j))F_c^\top)}\)$ where \(\varphi(\cdot)\) is a learnable adapter projecting patch embeddings to text dimension. \(M_\text{VLA}\) is supervised with Dice loss against ground-truth masks (\(\mathcal{L}_\text{VLA}\)). The localization map and VLA map are fused element-wise through a learnable network \(\psi(\cdot)\) for local-level scoring \(s_\text{VLA}\).

Final inference score: \(s(x') = \alpha s'_g + (1-\alpha)s'_\text{VLA}\).

Total loss: \(\mathcal{L}_\text{final} = \mathcal{L}_\text{loc} + \mathcal{L}_\text{VLA} + \mathcal{L}_G + \mathcal{L}_L\).

Key Contributions

  • VLAForge framework that jointly leverages visual forgery cues and cross-modal VLA semantics for DFD, rather than visual adaptation alone.
  • ForgePerceiver: independent forgery learner producing diversity-constrained forgery-aware masks (global) and a localization map (local), injected into CLIP without degrading pretrained VLA knowledge.
  • Identity-Aware VLA Scoring: identity-prior text prompting strategy that substitutes the <id> token with the frame's CLIP class token, producing fine-grained patch-level authenticity correspondence.
  • Comprehensive evaluation on 9 datasets (5 face-swapping + 5 full-face generation including GAN and diffusion models), both frame-level and video-level under cross-dataset protocol.

Results

  • Frame-level CDF-v2: VLAForge 87.0% AUROC vs. ForAda 84.3% (+2.7%) and RepDFD 77.3%.
  • Frame-level CDF-v1: VLAForge 93.9% AUROC vs. RepDFD 93.9% (second best) and CDFA 91.4%.
  • Video-level: up to 2.4% AUROC improvement over the best competing method across three datasets (CDF-v2, DFDC, DFD).
  • Full-face generation (DF40 subsets — VQGAN, StyleGAN-XL, SiT-XL/2, DiT, PixArt): VLAForge significantly outperforms RepDFD and ForAda on all five generative model subsets.
  • Ablation confirms both ForgePerceiver components (localization map +T1, forgery-aware masks +T2) and VLA Scoring each contribute incrementally across five datasets; baseline (frozen CLIP with handcrafted prompts) is substantially weaker.

Limitations

  • Requires pixel-level forgery mask annotations (\(G_x\)) for training (\(\mathcal{L}_\text{loc}\), \(\mathcal{L}_\text{VLA}\)), which are not always available or accurate for all forgery types.
  • The identity prior is derived from the frame's own class token embedding rather than an external identity reference, so it may not capture strong inter-frame identity consistency.
  • Inference speed and complexity analysis deferred to the appendix; the dual-branch design adds parameters over single-encoder baselines.
  • Evaluated in a fixed cross-dataset setting (train on FF++ c23); generalization to in-the-wild or highly compressed social-media videos is not assessed.
  • Full-face generation comparison is limited to two baselines (RepDFD, ForAda); broader comparison on those splits is absent.

Relevance to Vision-Language Models

VLAForge demonstrates a principled method for adapting CLIP's cross-modal alignment space — not just its visual encoder — to a discriminative downstream task, which is a significant departure from adapter-only VLM tuning paradigms. The identity-prior text prompting strategy is a concrete instantiation of conditioning language prompts on visual instance features at inference time, relevant to prompt engineering and visual-instruction-tuning research. The work highlights that patch-level vision–language alignment (rather than global cosine similarity) can surface fine-grained spatial correspondences useful for tasks requiring localization, broadening the applicability of VLMs beyond retrieval and classification. It also provides a model for combining independent auxiliary learners with frozen VLM backbones to preserve pretrained semantics while specializing for artifact detection.