Skip to content

SIF: Semantically In-Distribution Fingerprints for Large Vision-Language Models

🕒 Published (v1): 2026-04-18 15:52 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

In-distribution semantic fingerprints for VLM IP protection; CVPR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SIF is a non-intrusive ownership verification framework for Large Vision-Language Models (LVLMs) that embeds copyright fingerprints as semantically in-distribution trigger images, without modifying model parameters. It first exposes a practical attack (SDA) that defeats existing fingerprinting schemes by detecting their semantic anomalies, then counters it by distilling decoding-phase text watermark signals into the visual input while optimizing against worst-case representation perturbations.

Problem

Existing LVLM fingerprinting methods (IF, Proflingo, PLA) produce triggers or responses that are semantically out-of-distribution—high-perplexity prompt prefixes, adversarial images forcing unrelated outputs, or rare instruction pairs—making them easily detectable and removable. The paper formalizes this vulnerability: an adversary can deploy a lightweight reference LVLM, measure perplexity and semantic divergence of incoming queries, and silently substitute the stolen model's outputs with the reference model's, neutralizing all known fingerprints.

Method

Semantic Divergence Attack (SDA): A reference LVLM flags queries whose perplexity exceeds a threshold or whose output diverges (by non-stopword Jaccard similarity + all-MiniLM-L6-v2 cosine similarity) from the reference response, then replaces outputs—requiring no retraining, <5% false positives on normal traffic.

Semantic-Aligned Fingerprint Distillation (SAFD): Given a natural image \(x\) and prompt \(p\), a trigger image \(x' = x + \delta\) (\(\|\delta\|_\infty \le \varepsilon\)) is optimized so the frozen LVLM's decoding distribution, with standard (unmodified) decoding, matches a teacher watermarked distribution. The loss is: $\(\mathcal{L}(x') = \lambda_{\text{wm}} \mathcal{L}_{\text{wm}}(x') + \lambda_{\text{ce}} \mathcal{L}_{\text{ce}}(x')\)$ where \(\mathcal{L}_{\text{wm}}\) pushes probability mass toward the secret-key-derived watermark token list \(G_t\) (restricted to top-\(K\) tokens to stay in-distribution), and \(\mathcal{L}_{\text{ce}}\) cross-entropy-aligns the student with the teacher's watermarked response to prevent semantic drift. The underlying watermark scheme (Zhao et al.'s unigram method) assigns 50% of vocabulary to \(G_t\) per step with logit offset +4; detection uses a z-score test over generated tokens.

Robust-Fingerprint Optimization (RFO): To survive quantization/fine-tuning, RFO simulates worst-case embedding drift via a two-pass adversarial inner step: gradients of \(\mathcal{L}_{\text{base}}\) w.r.t. intermediate activations \(\{h_\ell\}\) define a norm-bounded perturbation \(\epsilon^\star_\ell\); the trigger is then optimized against the perturbed forward pass \(\tilde{h}_\ell = h_\ell + \epsilon^\star_\ell\).

Detection: the trigger image is queried to a suspect black-box API; if generated text's z-score exceeds a per-query threshold (calibrated so no unrelated model triggers it), ownership is verified. No model parameter access is required.

Key Contributions

  • Formalizes semantic vulnerability of existing fingerprints via the Semantic Divergence Attack (SDA): a training-free, low-overhead attack achieving near-zero FMR against all prior methods.
  • Proposes SAFD, which cross-modally distills decoding-phase text watermarks into trigger images, producing visually natural triggers and semantically coherent (in-distribution) responses under standard decoding.
  • Proposes RFO, a gradient-based worst-case representation perturbation that explicitly regularizes fingerprint robustness against quantization and fine-tuning.
  • Evaluates on LLaVA-1.5-7B and Qwen2.5-VL-7B against 7 baselines under 4-bit/8-bit quantization, 6+ fine-tuning datasets, decoding variations (sampling, paraphrasing), and image-level perturbations (JPEG, resize).

Results

Robustness under SDA (Table 3): - SIF achieves FMR of 0.86 (LLaVA-1.5-7B) and 0.85 (Qwen2.5-VL-7B) under SDA; all baselines (Proflingo, IF, PLA) drop to 0.

Robustness under quantization and fine-tuning (Table 2): - LLaVA-1.5-7B: SIF FMR 0.49 (4-bit), 0.89 (8-bit), 0.31–0.67 across 6 fine-tuning datasets; best baseline (PLA) reaches 0.40/0.79 on quantization, 0.00–0.49 on fine-tuning. - Qwen2.5-VL-7B: SIF FMR 0.88 (4-bit), 0.98 (8-bit), 0.43–0.89 across fine-tuning variants; consistently outperforms all baselines. - On LlavaMix (LLaVA full fine-tune), all baselines including PLA drop to 0.00–0.16; SIF achieves 0.31. - On Qwen GUI-Actor fine-tune: SIF 0.72 vs. best baseline PLA 0.14.

Stealthiness (Table 1): - SIF trigger PPL: avg 44.56, max 53.52 — comparable to normal queries (avg 75.7); Proflingo avg PPL 1261.3, IF avg 2467.4.

Robustness under output/input variations (Table 4, LLaVA-1.5-7B): - Paraphrasing: PLA 0.84, SIF 0.78; Sampling: PLA 0.86, SIF 0.80; JPEG compression: PLA 0, SIF 0.18; Resize: PLA 0, SIF 0.23.

Limitations

  • Trigger construction requires white-box access to the original LVLM (frozen, but accessible) — cannot verify ownership of truly closed-source models the developer does not hold.
  • RFO simulates embedding drift via gradient-based inner perturbation, which is an approximation; adaptive attacks specifically targeting the watermark token distribution are not evaluated.
  • FMR under aggressive fine-tuning remains meaningfully below 1.0 on several datasets (e.g., 0.31 on LlavaMix), leaving residual verification uncertainty.
  • Relies on the unigram watermarking scheme; sensitivity to watermark hyperparameters (e.g., vocabulary partition size, logit offset) is not fully ablated.
  • Input-space perturbations (JPEG, resize) reduce FMR noticeably (0.78→0.18), suggesting image preprocessing pipelines deployed by adversaries could weaken detection.

Relevance to Vision-Language Models

SIF directly addresses the practical IP protection problem that arises from widespread open-source LVLM release (LLaVA, Qwen-VL), which is central to the VLM ecosystem. By showing that semantically anomalous triggers—the dominant paradigm in LVLM fingerprinting—are trivially defeated, the paper motivates a fundamental redesign toward in-distribution verification. The cross-modal distillation of text watermarks into visual inputs is a novel bridge between content watermarking and model fingerprinting, relevant to researchers building trustworthy multimodal systems. The demonstrated robustness across quantization and fine-tuning variants tests exactly the deployment conditions common in the open VLM community.