Skip to content

AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; AdPO: preference optimization for adversarial robustness in LVLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AdPO reframes adversarial defense for Large Vision-Language Models (LVLMs) as a preference optimization problem, adapting DPO to make models prefer correct outputs on clean images while rejecting misleading outputs from adversarial images. Crucially, only the CLIP ViT image encoder is modified—not the LLM—enabling efficient small-to-large transfer and achieving state-of-the-art adversarial robustness without sacrificing clean performance.

Problem

Existing adversarial defenses for LVLMs fall into two paradigms: contrastive methods (e.g., FARE, TeCoA) that fine-tune the image encoder achieve computational efficiency but produce coarse-grained alignment; generative adversarial pre-training uses the full LVLM for finer alignment but causes significant clean-performance degradation. No prior work had applied preference optimization—already proven for LLM alignment—to adversarial robustness, despite the conceptual alignment between the two objectives.

Method

AdPO trains only the CLIP ViT encoder using a lightweight proxy model (TinyLLaVA: CLIP ViT-L/14 + OpenELM-450M-Instruct) and transfers the hardened encoder to larger LVLMs (LLaVA-1.5-7B) at inference.

Adversarial example generation. Adversarial images \(x_{adv}\) are produced via PGD (10 steps, \(\ell_\infty\), \(\varepsilon=2/255\)) by maximizing feature divergence: $\(x_{adv} = \arg\max_{\|x_{adv}-x_m\|_\infty \leq \varepsilon} \|\phi(x_{adv}) - \phi_{org}(x_m)\|_2^2\)$

Preferred Image Optimization (PIO). Online DPO using the model's own outputs: clean image responses serve as the winning sample \(y_w\), adversarial image responses as the losing sample \(y_l\)—no image annotations required: $\(\mathcal{L}_{PIO} = -\mathbb{E}\left[\log\sigma\left(\beta\log\frac{\pi_\theta(y_w|x_m,x_{text})}{\pi_{ref}(y_w|x_m,x_{text})} - \beta\log\frac{\pi_\theta(y_l|x_{adv},x_{text})}{\pi_{ref}(y_l|x_{adv},x_{text})}\right)\right]\)$

Adversarial Image Optimization (AIO). PIO alone fails to explicitly generate correct outputs under adversarial inputs and risks "unconditional preferences" (the model ignoring visual conditions). AIO applies dynamic SFT on adversarial inputs, weighting token-level loss by model confidence via a stop-gradient operator to prevent overfitting: $\(\mathcal{L}_{AIO} = -\mathbb{E}\left[\sum_{t=1}^{|y_l|} \text{sg}\!\left(\pi_\theta(y_w^t|y_w^{<t},x_{adv},x_{text})\right)\log\pi_\theta(y_w^t|y_w^{<t},x_{adv},x_{text})\right]\)$

Final objective: \(\mathcal{L}_{AdPO} = \mathcal{L}_{PIO} + \lambda\mathcal{L}_{AIO}\) with \(\lambda=1\), \(\beta=0.1\), AdamW, lr=1e-5, 2 epochs, batch size 128.

Key Contributions

  • First application of preference optimization (DPO) to adversarial robustness for LVLMs.
  • PIO+AIO dual strategy: PIO preserves clean performance; AIO mitigates unconditional-preference failure and explicitly improves adversarial accuracy.
  • Encoder-only training on a lightweight proxy (TinyLLaVA) with transfer to larger LVLMs, matching prior methods' compute while reducing overfitting risk.
  • SOTA results across image captioning (COCO, Flickr30k) and VQA (VQAv2, TextVQA) under untargeted adversarial attacks.

Results

Evaluated on LLaVA-1.5-7B with 100-step APGD untargeted attack; 500 images per dataset. Best prior baseline shown for each metric (Table 1):

  • COCO CIDEr (clean / \(\varepsilon\)=2/255 / \(\varepsilon\)=4/255): AdPO 115.3 / 68.9 / 47.6 vs. best prior 111.5 / 57.2 / 32.6 (AdvSimplex/AGH-MAT)
  • Flickr30k CIDEr: AdPO 75.9 / 38.6 / 27.9 vs. best prior 72.5 / 31.2 / 19.5 (AdvSimplex)
  • TextVQA accuracy: AdPO 35.5 / 24.2 / 17.6 vs. best prior 32.5 / 16.1 / 10.0 (Sim-CLIP / AGH-MAT)
  • VQAv2 accuracy: AdPO 73.6 / 52.3 / 37.6 vs. best prior 72.4 / 38.4 / 26.1 (Sim-CLIP / AdvSimplex)
  • Clean performance gap relative to undefended CLIP (115.5 CIDEr on COCO) reduced to <0.2 points, compared to drops of 4–17 points for all baselines.
  • Method also evaluated on Qwen2.5-VL and InternVL3.5 to demonstrate generalization beyond CLIP-based architectures (detailed results in appendix).

Limitations

  • Training exclusively on ImageNet without category labels; generalization to distribution-shifted or domain-specific datasets is unverified in the main text.
  • Encoder-only modification assumes CLIP ViT is the primary attack surface; non-CLIP LVLMs require separate validation (only partially addressed via Qwen2.5-VL experiments).
  • Evaluation subset is small (500 images per dataset) due to computational cost of 100-step APGD attacks, limiting statistical confidence.
  • The \(\lambda\) balancing PIO and AIO is set heuristically (\(\lambda=1\)); sensitivity analysis is deferred to appendix.
  • Targeted attack results (Table 2) are truncated in the provided text, so full coverage of the targeted-attack regime cannot be assessed here.

Relevance to Vision-Language Models

AdPO directly targets a critical deployment vulnerability in LVLMs: their inherited sensitivity of visual encoders (particularly CLIP ViT) to imperceptible adversarial perturbations. By adapting the DPO alignment objective—already central to modern LVLM post-training pipelines—for adversarial robustness, this work opens a new intersection between the safety-alignment and adversarial-robustness literatures. The encoder-only modification strategy is particularly significant for the VLM ecosystem, since a single hardened CLIP ViT can be dropped into any CLIP-based LVLM (LLaVA, OpenFlamingo, BLIP-2) without retraining the language backbone. This positions AdPO as a practical plug-in robustness upgrade complementary to ongoing work on VLM hallucination, multimodal alignment, and safety fine-tuning.