Skip to content

On Adversarial Vulnerability of Vision-Language Models through the Lens of Intermediate Spectral Subspaces

🕒 Published (v1): 2026-07-08 13:06 UTC · Source: Arxiv · link

Why this paper was selected

Spectral subspace lens on VLM adversarial vulnerability; novel mechanistic robustness analysis

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper investigates adversarial vulnerability in transformer-based VLMs through the spectral structure of intermediate linear transformations, specifically the bottom right singular-vector subspaces. The authors show that untargeted adversarial optimization naturally steers intermediate representations toward these information-attenuating subspaces and propose SSGRA, a white-box attack that explicitly exploits this mechanism. SSGRA outperforms six representative baselines on Gemma-3, Qwen2.5-VL, and LLaVA-1.5.

Problem

Prior adversarial robustness theory for DNNs focuses on input-space geometry, decision boundaries, non-robust features, Jacobians, and Lipschitz properties of large singular values — but the spectral behavior of intermediate linear transformations and, in particular, the role of near-null (bottom) right singular-vector subspaces as an attack surface remains unexplored. For transformer-based VLMs, where numerous learnable projection matrices appear in self-attention, FFNs, and fusion modules, this leaves a mechanistic gap in understanding how adversarial signals propagate.

Method

The paper proposes the Spectral Subspace Guided Representation Attack (SSGRA). For each weight matrix \(W \in \mathbb{R}^{m \times n}\) with SVD \(W = U\Sigma V^\top\), the bottom-\(s\) right singular vectors \(V_s^{\mathrm{bottom}} = \{v_{n-s+1}, \ldots, v_n\}\) span a subspace where input energy is most attenuated. The key spectral alignment measure for a set of token representations \(H = \{h_1,\ldots,h_N\}\) is:

\[\Psi_s(H, V_s^{\mathrm{bottom}}) = \frac{1}{N}\sum_{i=1}^{N}\sum_{j=n-s+1}^{n}\left(\frac{v_j^\top h_i}{\|h_i\|_2}\right)^2\]

SSGRA solves a joint maximization over the \(L_\infty\)-constrained perturbation set \(B_c^\infty(x)\):

\[x_a^* = \arg\max_{x_a \in B_c^\infty(x)} \left\{ -\lambda \left[\sum_{k=1}^K \sum_j \cos(h_k^{v,(j)}(x), h_k^{v,(j)}(x_a)) + \sum_{\ell=1}^L \sum_i \cos(H_\ell^{(i)}(x), H_\ell^{(i)}(x_a))\right] + (1-\lambda)\sum_{m\in\mathcal{S}}\Psi_s(z_m(x_a), V_{m,s}^{\mathrm{bottom}})\right\}\]

The first term (from BSA) maximizes block-wise cosine discrepancy across all visual encoder and LLM layers; the second maximizes alignment of pre-transformation representations with bottom singular subspaces at a selected subset of layers \(\mathcal{S}\). Layer selection is done via validation-set ablation. Optimization uses Adam with 1000 gradient steps. A complementary layer-wise probing experiment attacks individual layers and measures resulting spectral alignment shifts to verify that standard untargeted optimization already drifts toward bottom subspaces without explicit enforcement.

Key Contributions

  • Identifies bottom right singular-vector subspaces of intermediate linear transformations as a previously overlooked adversarial attack surface in transformer-based VLMs.
  • Demonstrates empirically that standard untargeted adversarial optimization (BSA, which has no spectral objective) naturally increases alignment with bottom singular subspaces at MLP gate/up-projection layers, without explicit enforcement.
  • Proposes SSGRA, a spectrally guided white-box attack combining representation discrepancy and subspace alignment objectives.
  • Provides a spectral characterization of Qwen2.5-VL, LLaVA-1.5, and Gemma-3, showing that near-null singular directions (\(\sigma_{\min} < 10^{-3}\)) occur in 24.90%, 27.67%, and 12.22% of non-attention operators, respectively, correlating with attack effectiveness.
  • Links VLM adversarial vulnerability to the instability of ill-posed inverse problems via the near-null singular direction analogy.

Results

  • Qwen2.5-VL: SSGRA achieves 7.90–19.74% additional relative degradation over the strongest baseline (BERTScore F1) and 30.50–97.93% (ROUGE-L F1), across \(\ell_\infty\) budgets \(c \in [0.002, 0.005]\).
  • LLaVA-1.5: Up to 7.46% (BERTScore F1) and 18.90% (ROUGE-L F1) additional relative degradation over the strongest baseline; gains increase with budget.
  • Gemma-3: Up to 0.80% (BERTScore F1) and 3.07% (ROUGE-L F1) additional relative degradation; smaller margins consistent with Gemma-3 having fewer near-null singular directions (~half those of Qwen2.5-VL/LLaVA-1.5).
  • Models tested on 100 ImageNet images per setting with the prompt "What is shown in the image?"; evaluated via BERTScore (RoBERTa-large) and ROUGE-L Precision/Recall/F1; lower scores indicate stronger attacks.
  • Baselines: BSA, DRA, FDA, SSPA, EGA, NLL/CE — SSGRA outperforms all across all models and budgets.

Limitations

  • Layer selection for \(\mathcal{S}\) requires a held-out validation set; no adaptive or gradient-free selection method is provided, limiting plug-and-play applicability.
  • Evaluation restricted to 100 images per condition; statistical robustness at larger scale is unverified.
  • Only untargeted white-box attacks are studied; transferability (black-box) and targeted settings are not explored.
  • The analogy to ill-posed inverse problems is qualitative; a formal bound connecting bottom-subspace alignment to attack success rate is absent.
  • The increased computational cost of SSGRA (SVD computation per layer) relative to baselines is noted but Table 2 with full FLOP counts is truncated in the provided text.
  • Spectral alignment dynamics are studied primarily on Gemma-3; generalization of the layer-wise probing findings to all three models is not fully shown.

Relevance to Vision-Language Models

This work opens a new mechanistic angle on VLM adversarial robustness by connecting it to the spectral conditioning of internal weight matrices rather than input-space geometry or end-to-end Jacobians — a perspective directly motivated by the multi-component architecture (visual encoder → fusion → LLM) unique to VLMs. The finding that near-null singular directions are far more prevalent than large singular values in contemporary VLMs (Qwen2.5-VL, LLaVA-1.5, Gemma-3) implies that current spectral regularization practice (constraining large singular values) may leave a dominant vulnerability mode unaddressed. For VLM researchers, this suggests that bottom-subspace regularization or singular-value conditioning during training could be a tractable robustness intervention, and that the proportion of near-null operators serves as a predictive spectral metric for adversarial susceptibility.