Skip to content

Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Standard adversarial attacks overfit to the original model and fail after fine-tuning, making copyright tracking of LVLMs unreliable. This paper proposes Parameter Learning Attack (PLA), which simultaneously updates both the trigger image and the model parameters (in opposing directions) during adversarial optimization, forcing the trigger to generalize across fine-tuned derivatives. PLA requires no modification to the published model and operates post-release in a black-box verification regime.

Problem

When a published LVLM is fine-tuned by a third party (legally or illicitly), vanilla adversarial trigger images—constructed on the original model—fail to elicit the target response from the derivative model because adversarial perturbations overfit to the original parameter set. Existing LLM copyright methods (backdoor/fingerprint injection) require modifying and retraining the model before release, which degrades performance and consumes large compute. No prior work addressed copyright tracking specifically for LVLMs, whose visual modality provides a novel attack surface.

Method

PLA constructs a trigger image via a minimax adversarial optimization. During each PGD iteration: 1. Model parameters are updated to maximize the cross-entropy loss between the model output and the trigger target (simulating fine-tuning resistance): Θ' ← Θ' + β · clip(∇_Θ' L). 2. Trigger image pixels are updated to minimize the same loss (adversarial attack): x' ← x' − α · sign(∇_x' L), with perturbation clipped to ε = 16/255.

The competing dynamics force the trigger to overcome artificially introduced model resistance, producing perturbations that generalize to models whose parameters have shifted via real fine-tuning. Trigger design uses rare, semantically incoherent QA pairs (e.g., "Q: Please stop. A: I'm playing games.") absent from standard fine-tuning datasets, preventing accidental memorization. Copyright verification computes the Target Match Rate (TMR) by querying suspicious models with the trigger set under black-box access only.

Key Contributions

  • First study on copyright tracking specifically for LVLMs.
  • PLA: post-release trigger construction that does not alter the published model's parameters.
  • A minimax adversarial learning dynamic that simulates fine-tuned model behavior during trigger construction.
  • Empirical demonstration across 6 downstream VQA datasets (V7W, ST-VQA, TextVQA, PaintingForm, MathV360k, ChEBI-20) under both LoRA and full fine-tuning.
  • Robustness analysis against input transformations (noise, Gaussian/mean blur) and model-level defenses (weight pruning, parameter perturbation).

Results

  • PLA vs. baselines (Table 1, average TMR):
  • LoRA fine-tuning: PLA 52%, RNA 23%, IF 18%, Ordinary 3%.
  • Full fine-tuning: PLA 54%, RNA 17%, IF 12%, Ordinary 2%.
  • PLA achieves the best TMR on all 6 fine-tuned models under both fine-tuning regimes.
  • Best single QA pair ("Q: Please stop. A: I'm playing games."): average TMR 72% (LoRA), 66% (FFT).
  • Robustness to input transformations (Table 3): TMR drops ~10–28 pp under noise/blur but remains far above Ordinary attack baseline.
  • Robustness to model pruning/perturbation (Table 4): Combined pruning (attention+MLP) reduces TMR by 15–22 pp; model perturbation causes smaller drops (10–15 pp).
  • Unrelated models (MiniGPT-4, Qwen2-VL) show no response to triggers, confirming specificity.

Limitations

  • TMR on MathV360k fine-tuned models is consistently low regardless of QA pair, suggesting domain-specific fine-tuning can partially erode trigger efficacy.
  • Model learning rate β in PLA requires tuning: too low → insufficient generality; too high → trigger fails to converge.
  • Robustness degrades non-trivially under pixel-level noise (up to 28 pp TMR drop), indicating triggers are not fully robust to active input preprocessing attacks.
  • Evaluated only on LLaVA-1.5; generalization to other LVLM architectures (e.g., InternVL, Qwen2-VL) is not empirically verified.
  • Stealers with white-box access could potentially identify and filter rare QA patterns from fine-tuning data, though the paper argues such pairs are absent from standard datasets.

Relevance to Vision-Language Models

This work directly addresses the governance and IP-protection side of LVLM deployment, which is increasingly relevant as powerful open-weight VLMs (LLaVA, InternVL, Qwen-VL) are released for community fine-tuning. PLA exploits the visual modality as a unique channel for watermarking that has no LLM analogue, demonstrating that adversarial images can serve dual-use purposes beyond attacks. For researchers tracking VLMs, this paper establishes the threat model and first baseline for LVLM copyright tracking, and its minimax formulation connects to broader questions about adversarial transferability and parameter sensitivity in large multimodal models.