Skip to content

Rethinking Bottlenecks in Safety Fine-Tuning of Vision Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; identifies fundamental bottlenecks in VLM safety fine-tuning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing safety fine-tuning methods for VLMs fail on challenging multi-image scenarios because they lack safety visual reasoning — the ability to infer harmful intent from the relationship between two individually benign images. This paper introduces MIS, the first multi-image safety dataset (4k train / 2185 test), and MIRage, a fine-tuning method using safety Chain-of-Thought labels, which reduces Attack Success Rate to near zero while improving general benchmark performance.

Problem

Single-image and text-only safety SFT methods (e.g., VLGuard, Textual SFT) face two bottlenecks: (1) over-conservatism — models refuse benign inputs because SFT labels are dominated by simple "I'm sorry" rejections; and (2) failure on compositional multi-image attacks where harmful intent arises from the interaction of two safe images combined with a neutral text query, not from any individual unsafe element.

Method

Dataset construction (MIS): A four-step automated pipeline generates multi-image-text triples where harmful intent is encoded in image–image relationships: 1. Extract unsafe semantic elements from existing safety benchmarks using Qwen2.5-72B and InternVL2.5-78B. 2. Generate detoxified neutral text instructions referencing two objects ("the tool in the first image…"). 3. Auto-refine image generation with Stable Diffusion 3.5 Large, using InternVL2.5-78B to refine text-to-image prompts for better object–text alignment. 4. Human experts + GPT-4o filter and classify into training (text contains unsafe intent) and three test splits: MIS-easy (explicit unsafe visual elements), MIS-hard (no explicit unsafe elements; harmful only via cross-image reasoning), and MIS-real (real images from LAION2B).

MIRage fine-tuning: InternVL2.5-8B is fine-tuned on the 4k MIS training set with safety CoT labels generated by InternVL2.5-78B. The CoT labels guide the model to (a) describe visual content of both images, (b) reason about how their combination implies harmful intent, and (c) produce a safe, contextually appropriate response. 500 general QA samples from M4-Instruct are mixed in (11% of the training set).

Key Contributions

  • Identification and analysis of two SFT bottlenecks: over-prudence caused by simplistic refusal labels, and failure to reason about cross-image harmful intent.
  • MIS: first multi-image safety dataset with 4k training samples and three-tier test splits (easy/hard/real) covering 6 unsafe categories and 12 subcategories.
  • Empirical finding that simple CoT label reconstruction (VLGuard-R) already recovers general performance lost by VLGuard-P, motivating safety CoT for multi-image settings.
  • MIRage: fine-tuning recipe that achieves near-zero ASR on MIS while raising average accuracy by 0.83% across five general benchmarks — eliminating the safety–helpfulness trade-off.

Results

  • MIS test set (InternVL2.5-8B + MIRage): ASR reduced from 80.12% → 0.24% (easy), 84.51% → 0.20% (hard), 76% → 0% (real); Reasoning Success Rate (RSR) reaches 99.34% / 99.80% / 100% vs. base model's ~10–15%.
  • General benchmarks: +0.83% average accuracy across MMStar, MMMU (SI&MI), MMT-Bench-MI vs. base InternVL2.5-8B.
  • FigStep ASR (single-image): base 38.80% → MIRage 0.60%, matching VLGuard-R.
  • MM-SafetyBench ASR: 15.58% (base) → 0.54% (MIRage), best among all methods.
  • MSSBench Unsafe Acc: base 3.00% → MIRage 40.00%, surpassing VLGuard-R (35.44%) and Textual SFT (1.00%).
  • SIUO Safe Acc: 24.85% (base) → 71.26% (MIRage), largest improvement.
  • GPT-4o achieves ASR of 46.21%/65.29% (easy/hard) and Gemini-1.5-pro ~40% on both, showing vulnerability of SOTA closed-source models to multi-image attacks.

Limitations

  • Training and main experiments are centered on InternVL2.5-8B; generalization across architectures is tested only in appendices.
  • MIS-easy and MIS-hard rely on T2I-generated synthetic images (Stable Diffusion 3.5 Large), which may differ in distribution from real-world attack images.
  • MIS-real is small (200 images) and constructed via retrieval matching, limiting coverage of real-world multi-image attack diversity.
  • The safety CoT label generation depends on InternVL2.5-78B, introducing noise and potential labeling errors not fully characterized.
  • Only 11% general QA samples in training may not suffice to preserve instruction-following across all domains at scale.

Relevance to Vision-Language Models

This paper directly targets a fundamental alignment challenge for VLMs: single-image safety training does not transfer to multi-image inputs, a gap that grows as VLMs increasingly handle interleaved or multi-image contexts. The safety visual reasoning framework — using CoT labels that explicitly link cross-image semantics to harmful intent — offers a principled approach complementary to RLHF and standard SFT. For researchers tracking VLMs, MIS provides both a challenging evaluation suite and a training recipe that decouples harmlessness from helpfulness, challenging the common finding that safety alignment degrades general capability. The work also benchmarks frontier models (GPT-4o, Gemini-1.5-pro) and reveals that multi-image jailbreaks remain an open vulnerability even for closed-source APIs.