Skip to content

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

🕒 Published (v1): 2026-07-20 17:58 UTC · Source: Arxiv · link

Why this paper was selected

Domain-generalized pixel-level tampering detection in VLMs; practical safety/integrity tooling

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PIXAR-DG is a domain-generalized training framework for pixel-level image tampering detection that must remain robust against VLMs it has never seen. Two practical strategies—balanced real/tampered minibatch sampling and a late-injection schedule for small new-domain data—yield a 26.1%/26.8% relative improvement in average gIoU/cIoU over prior SOTA (PIXAR), while using only 19.2% of the original training data.

Problem

Detectors trained on images manipulated by one VLM (e.g., Qwen-Image) overfit to model-specific synthesis artifacts and fail under cross-generator distribution shift, because modern VLMs (GPT-Image-2.0, Gemini, FLUX, Seedream) differ substantially in architecture, generation pipeline, and post-processing. No prior work addressed out-of-distribution generalization for pixel-level tampering localization—only image-level classification—leaving a critical forensic gap as new closed-source models continuously appear.

Method

The method builds on the PIXAR architecture (CLIP vision encoder + SAM mask decoder + LLM backbone, LoRA rank \(r=8\), \(\alpha=16\)) and introduces three coordinated training modifications.

1. Balanced minibatch sampling. Given a multi-domain dataset \(\mathcal{D} = \bigcup_{k=1}^{K} \mathcal{D}^{(k)}\) where each domain contains real and tampered subsets, naĂŻve uniform pooling produces an effective distribution proportional to dataset size, causing tampered-image dominance. Instead, each minibatch of size \(B\) is explicitly split: $\(B_{\text{real}} = \lfloor \rho B \rfloor, \quad B_{\text{tamp}} = B - B_{\text{real}},\)$ with \(\rho = 0.5\) (equal ratio), inducing a balanced training distribution \(p_{\text{bal}}(c) \in \{\rho, 1-\rho\}\) and preventing optimization collapse toward either class.

2. Late-injection training. New-domain data \(\mathcal{D}_{\text{new}}\) (small, \(\ll \mathcal{D}_{\text{base}}\)) is withheld for the first \(T_{\text{inj}}\) steps. The model first trains only on \(\mathcal{D}_{\text{base}}\) to learn stable representations, then the late-stage objective becomes: $\(\mathcal{R}_{\text{late}}(\theta) = (1-\alpha)\,\mathbb{E}_{x \sim \mathcal{D}_{\text{base}}}[\ell(f_\theta; x)] + \alpha\,\mathbb{E}_{x \sim \mathcal{D}_{\text{new}}}[\ell(f_\theta; x)],\)$ preventing the scarce new-domain samples from biasing early feature learning.

3. Reduced constant learning rate. The learning rate is reduced from \(1 \times 10^{-4}\) to \(2 \times 10^{-5}\) with a constant schedule (replacing warmup-then-decay), reducing overfitting to source-domain artifacts.

The multi-task loss combines five terms: \(\mathcal{L}_{\text{total}} = \lambda_{\text{sem}}\mathcal{L}_{\text{sem}} + \lambda_{\text{bce}}\mathcal{L}_{\text{bce}} + \lambda_{\text{dice}}\mathcal{L}_{\text{dice}} + \lambda_{\text{cls}}\mathcal{L}_{\text{cls}} + \lambda_{\text{text}}\mathcal{L}_{\text{text}}\), covering semantic label classification, pixel-level BCE and DICE localization, global real/tampered detection, and natural-language explanation generation.

Training uses 70K Qwen-Image samples (selected empirically as the best-generalizing base source) plus 3K Gemini-2.5 samples (the base model's worst OOD case, making it the highest-leverage companion source), totaling 73K images on 4Ă— RTX A6000 GPUs.

Key Contributions

  • First formal study of domain generalization for pixel-level image tampering detection across modern VLMs (termed PIXAR-DG task).
  • Balanced real/tampered minibatch sampling that prevents training collapse under imbalanced multi-domain pooling.
  • Late-injection optimization enabling safe absorption of scarce new-domain data without early representation bias.
  • Empirical source-selection analysis showing Qwen-Image has the highest cross-generator OOD average accuracy (60.7%, +32.4% over second-best Gemini-2.5) and that adding more than 70K base samples yields no further OOD gain due to companion-source dilution.
  • State-of-the-art localization and classification results using only 19.2% of the PIXAR training set.

Results

  • 7B models — OOD average (gIoU / cIoU): PIXAR-DG-7B: 0.193 / 0.201 vs. PIXAR-7B: 0.159 / 0.166; +21.4% / +21.1% relative improvement.
  • 13B models — OOD average (gIoU / cIoU): PIXAR-DG-13B: 0.222 / 0.232 vs. PIXAR-13B: 0.176 / 0.183; +26.1% / +26.8% relative improvement.
  • Pixel Recall (7B): 45.11% vs. PIXAR-7B 25.77% (+19.34% absolute); Pixel F1 (7B): 33.29% vs. 28.49% (+4.80% absolute).
  • Largest single-generator gain (7B): FLUX.2 gIoU 0.170 → 0.222 (+30.6% relative).
  • Binary OOD classification accuracy: PIXAR-DG-7B 79.6% vs. PIXAR-7B 69.6% (+10.0% absolute); PIXAR-DG-13B 84.0% vs. PIXAR-13B 59.0% (+25.0% absolute).
  • Compared against LISA-7B/13B, SIDA-7B/13B, PIXAR-7B/13B; PIXAR-DG is best on every averaged OOD metric.

Limitations

  • Companion-source leakage boundary: 3K Gemini-2.5 training samples are drawn from the Gemini-2.5 test pool held-out partition, which partially blurs the train/test separation for that generator.
  • Base-source data scaling saturation: Increasing Qwen-Image beyond 70K degrades OOD performance (380K full-set average OOD accuracy 63.4% < 70K's 79.6%), implying companion-source dilution—the method is sensitive to the relative proportion of base vs. new-domain data.
  • Source selection requires empirical search: Choosing the best base and companion generator currently requires training six single-source variants; no principled criterion is provided for unseen generator sets.
  • Architecture dependency: The framework is validated only on PIXAR's CLIP+SAM+LLM backbone; generality to other segmentation architectures is unverified.
  • Closed-source generator access: New-domain data for late injection requires being able to obtain edited image pairs from the target VLM, which may not be feasible for fully closed systems.

Relevance to Vision-Language Models

This paper is directly relevant to anyone tracking VLMs because it treats modern VLMs (GPT-Image, Gemini, Qwen, FLUX, Seedream) as adversarial generation domains whose outputs must be forensically authenticated—an increasingly critical problem as VLM image-editing fidelity improves. The detector itself is a multimodal VLM (CLIP + SAM + language decoder with LoRA) that must generalize across the distributions created by other VLMs, framing the forensics problem as a cross-VLM domain generalization challenge. The finding that Qwen-Image uniquely provides broadly transferable artifact representations—with 60.7% cross-generator OOD accuracy versus ≤28.3% for all other single sources—suggests meaningful structural differences in how open-source versus closed-source VLMs encode manipulation traces. The work also demonstrates that VLM-based perceptual detectors can be adapted to unseen generator distributions with as little as 3K new samples via curriculum scheduling, a practical insight for rapid forensic response to newly released models.