Skip to content

Misaligned Roles, Misplaced Images: Structural Input Perturbations Expose Multimodal Alignment Blind Spots

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; structural input perturbations reveal multimodal alignment blind spots

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multimodal alignment training focuses on the assistant role and assumes a fixed prompt structure, leaving VLMs brittle to structural perturbations. This paper introduces Role-Modality Attacks (RMA), which exploit role-swap (user↔assistant) and image-token repositioning to bypass safety refusals without modifying query content. An adversarial training procedure over all eight structural configurations nearly eliminates the attack success rate while preserving model utility.

Problem

Post-training safety alignment of VLMs (via RLHF, preference tuning) targets only the assistant role and conditions on a single canonical chat template. The user role remains largely unaligned, and shifting the <|image|> token from its default position creates out-of-distribution inputs that existing defenses do not cover—a structural blind spot orthogonal to content-based attacks.

Method

Role Confusion: The user and assistant role tags in the chat template are swapped, forcing the model to generate from the user perspective, which is less aligned.

Modality Manipulation: The image token is moved from its default position to (1) removed entirely, (2) default (img pos), (3) end of user turn (img end), or (4) start of assistant turn (img out), inducing distributional shift that disrupts learned refusal behavior.

Attack Composition: The two attack types are combined, yielding 8 configurations (\(4 \text{ modality positions} \times 2 \text{ role states}\)), which compose super-additively.

Activation-Space Analysis: Refusal features \(r_\text{RF}^{(l)}\) are extracted via difference-in-means on AdvBench vs. Alpaca residual stream activations. Attack vectors \(r_A^{(l)}\) (mean shift from original to adversarial successful prompts) are compared to \(-r_\text{RF}^{(l)}\) via cosine similarity and projection: $\(\text{proj}_{-r_\text{RF}^{(l)}} r_A^{(l)} = \frac{r_A^{(l)} \cdot (-r_\text{RF}^{(l)})}{\|-r_\text{RF}^{(l)}\|^2}(-r_\text{RF}^{(l)})\)$ The projection coefficient (not cosine similarity alone) explains compositional strength gains.

Adversarial Training: QLoRA fine-tuning on ~10k harmful+harmless prompts, each perturbed with all 8 RMA configurations, minimizing cross-entropy toward refusal (harmful) and normal response (harmless) under all structural variants.

Key Contributions

  • Identifies a structural alignment blind spot in VLMs: user-role misalignment and fixed modality token position dependency.
  • Introduces RMA—a computationally lightweight, query-agnostic attack class operating purely through chat-template manipulation.
  • Demonstrates super-additive compositionality of Role Confusion and Modality Manipulation attacks, and further composition with content-based attacks (AIM, AutoDAN, GCG, etc.).
  • Provides mechanistic explanation via refusal-feature projection in the residual stream, distinguishing it from cosine similarity alone.
  • Proposes an adversarial training mitigation (QLoRA over 8 structural variants) that near-eliminates ASR while maintaining VQA-V2 accuracy.

Results

  • Qwen2-VL-7B: Composed attack img out_swap reaches ASR of 42.50% (TS) on AdvBench; individual attacks are 5–33%. After adversarial training: 0.00% ASR across all settings.
  • LLaVA-1.5-7B: Composed attacks reach up to 97.12% ASR (TS) on AdvBench; after AT reduced to ≤0.38% (TS).
  • Phi-3.5-vision: Composed attacks reach 80.00% ASR (TS) on AdvBench; after AT reduced to <4%.
  • HarmBench: Similar patterns hold; e.g., LLaVA img out_swap at 97.50% (TS) before AT, 3.00% after.
  • Utility preservation: VQA-V2 accuracy essentially unchanged post-AT (e.g., Qwen: 81.53%→81.73%); benign Alpaca refusal rate remains low (≤6.73%).
  • Projection onto \(-r_\text{RF}\) for composed attacks peaks near 1.0 at intermediate layers, approaching the strength of the full negative refusal direction.

Limitations

  • Adversarial training dataset uses only text-paired harmful/harmless instructions; image content is randomized but not systematically varied across harmful visual content types.
  • Mitigation evaluated on three base models; generalization to larger or newer VLMs (e.g., 70B+) is addressed only partially in the appendix.
  • Cosine similarity and projection analyses are performed on 7B-scale models; interpretability findings may not transfer to models with different architectural depth or width.
  • RMA assumes white-box knowledge of the chat template format, which is typically public but may evolve.
  • Adaptive attacks (adversaries who know the adversarial training procedure) are not fully evaluated.

Relevance to Vision-Language Models

This work directly targets the safety alignment pipeline of instruction-tuned VLMs, revealing that standard RLHF/preference-tuning practice creates role-asymmetric and structure-sensitive models. For researchers tracking VLMs, the finding that modality token position alone can bypass refusal mechanisms highlights a previously underexplored attack surface specific to multimodal (as opposed to text-only) models. The mechanistic analysis linking structural perturbations to the refusal-feature direction in the residual stream connects to broader interpretability work on how safety is encoded in transformer activations. The adversarial training recipe provides a practical, PEFT-compatible mitigation applicable during post-training alignment of any VLM.