Skip to content

VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection

🕒 Published (v1): 2026-07-07 13:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VendorBench-100 evaluates 36 deepfake image detectors — 5 commercial APIs, 7 zero-shot vision-language models (VLMs), and 24 open-source classifiers — on a single adversarial 100-image corpus under a unified output schema. The central finding is a systematic divergence between ranking power (ROC-AUC) and operating-point quality (MCC), meaning strong class discrimination does not imply reliable default-threshold decisions. Commercial APIs lead by median MCC, VLMs occupy the middle tier, but individual open-source detectors match or exceed VLMs on pure ranking power.

Problem

Deepfake detection is served by three paradigms — commercial APIs, zero-shot VLMs, and open-source detectors — that are almost never evaluated on common ground. Practitioners choosing between them have no single, independently controlled comparison. Existing academic benchmarks test trainable architectures on large homogeneous corpora, not closed APIs or general-purpose VLMs, and they rarely reflect adversarial, real-world edge cases. Standard accuracy metrics are also trivially gamed under class imbalance (a naïve always-fake predictor scores 79% accuracy yet MCC = 0.00).

Method

The authors assemble a fixed 100-image corpus (79 fake, 21 real; 79:21 skew) curated around eight edge-case families: live face-swap motion smearing, near-duplicate online face swaps, letterboxed text-to-video stills, avatar compositing seams, fully synthetic text-to-image output, on-device AI photo edits of real scenes, opaque-provenance manipulations, and heavily compressed research-dataset frames. 78 of 100 images are ≤256 px on the long side, eroding high-frequency cues.

All 36 models are served images under neutral numeric filenames (anti-leakage protocol) and their heterogeneous outputs — categorical tags, confidence scores, or free-text verdicts — are normalized into a canonical record: hard label (FAKE/REAL), a scalar \(\hat{p} \in [0,1]\) interpreted as \(P(\text{fake})\), and a success flag encoding abstentions. Primary ranking metric is MCC:

\[\text{MCC} = \frac{TP \cdot TN - FP \cdot FN}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}}\]

ROC-AUC is reported as a threshold-independent tiebreak. Coverage (fraction of decided images) is tracked separately to handle abstentions under a reject-option framework.

Key Contributions

  • VendorBench-100 corpus: 100 deliberately hard images across 8 edge-case families, 21 provenance sources, with per-image provenance registry and anti-leakage filename protocol.
  • Cross-paradigm unified schema: normalizes commercial API outputs, VLM free-text verdicts, and open-source classifier scores into one comparable record.
  • 36-model leaderboard: first direct comparison of all three paradigms under identical metrics and corpus.
  • MCC vs. ROC-AUC divergence finding: identifies recurring "danger zone" models with high AUC but negative MCC — good discriminators with badly miscalibrated default thresholds — as the central empirical result rather than a single-paradigm ranking.
  • Full evaluation harness, per-image evidence, and benchmark results released publicly.

Results

  • Commercial APIs achieve the strongest median MCC, followed by VLMs, then most open-source detectors.
  • Best open-source detector surpasses every VLM in the study on ranking power (ROC-AUC).
  • Leaderboard paradox: TruthScan posts the highest ROC-AUC (0.915) yet ranks 32nd of 36 by MCC (−0.13), illustrating that threshold calibration dominates operating-point quality.
  • Divergence between ROC-AUC and MCC recurs across all three paradigms, not restricted to any single model class.
  • Naïve always-fake baseline: 79% accuracy, 0.83 F1, but MCC = 0.00 — motivating the metric choice.

Limitations

  • Corpus size is only 100 images; per-group sample sizes across 21 source families are small, precluding calibrated field-accuracy claims.
  • Single-run design (no repeated sampling or confidence intervals on MCC/AUC).
  • 79:21 class imbalance is intentional but limits symmetry of class-conditional error analysis.
  • Open-source detectors were not trained on the specific generator mix in the corpus, so degraded performance reflects distribution shift as much as inherent model quality.
  • Anti-leakage protocol prevents filename/metadata exploitation but cannot rule out pretraining data contamination for VLMs.
  • Commercial API internals are opaque; failure-mode attribution is not possible.

Relevance to Vision-Language Models

This benchmark provides the first controlled head-to-head evaluation of zero-shot VLMs (GPT-4V, Gemini, Claude Opus, Qwen-VL, Llama 4 Maverick, Nemotron, GLM) as deepfake detectors against both commercial and open-source alternatives, quantifying the practical cost of lacking forgery-specific training. The finding that VLMs occupy a middle tier — inferior to commercial APIs by median MCC but occasionally matched by open-source detectors in ranking power — directly informs whether practitioners can repurpose existing VLM API access for content moderation without specialized fine-tuning. The divergence between ROC-AUC and MCC is particularly salient for VLMs, which generate free-text verdicts normalized post-hoc, making threshold calibration an additional pipeline-design concern not faced by purpose-built classifiers.