Get RICH or Die Scaling: Profitably Trading Inference Compute for Robustness

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Inference-time compute improves OOD robustness in VLMs; ICLR 2026 analysis

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes the Robustness from Inference Compute Hypothesis (RICH): inference-compute defenses against adversarial attacks become more effective as a model's training data better reflects components of the attacked data. Experiments across VLMs with varying adversarial training levels confirm a "rich-get-richer" dynamic where test-time compute scaling yields robustness gains only when the base model is already sufficiently robust. By lightweight adversarial finetuning of InternVL 3.5 gpt-oss 20B's vision encoder, the authors create the first adversarially robust RL-tuned reasoning VLM.

Problem

Prior work (Zaremba et al., 2025) showed that scaling inference-time compute (extended reasoning) reduces text jailbreak success rates, but this benefit degrades to near-zero under gradient-based or multimodal vision attacks — even strong reasoning models like o1-v sustain a 39% attack success rate on Attack-Bard at maximum inference compute. The conditions under which inference scaling grants adversarial robustness were unknown, limiting practical benefit.

Method

The authors formalize the RICH: the closer attacked data's components are to model training data, the more test-time compute aids robustness, mediated by compositional generalization — the model's ability to follow security specifications on adversarially out-of-distribution (OOD) inputs by composing from in-distribution components.

Three experiment protocols test the RICH: 1. Security specification sufficiency (Section 4.1): White-box PGD attacks (\(\varepsilon = 16/255\), \(\ell_\infty\), 300 steps) on visual prompt injection against LLaVA-v1.5, FARE-LLaVA-v1.5, and Delta2-LLaVA-v1.5 with and without explicit security specifications; response pre-filling verifies whether specification tokens alone drive robustness. 2. Inference compute scaling (Section 4.2): Naive scaling by repeating the security specification \(K \in \{0,1,3\}\) times against PGD (\(\varepsilon = 64/255\), 100 steps) across the same three LLaVA-style VLMs; also tests reduced \(\varepsilon = 16/255\) to verify that bringing attacked data closer to training data enables weaker models to benefit. 3. Black-box attacks + reasoning VLMs (Section 4.4): Attack-Bard (black-box transferred attacks) on InternVL 3.5 gpt-oss 20B with and without lightweight unsupervised adversarial finetuning of its ViT (Schlarmann et al., 2024), measuring adversarial accuracy as reasoning tokens scale from 1 to 2048.

Model robustness spectrum: LLaVA-v1.5 (no adversarial training) → FARE-LLaVA-v1.5 (CLIP encoder replaced with adversarially finetuned version, \(\varepsilon = 2/255\)) → Delta2-LLaVA-v1.5 (full adversarial contrastive pretraining + adversarial visual instruction tuning, \(\varepsilon = 8/255\)).

Key Contributions

• Proposes the RICH as a unifying explanatory hypothesis for when inference-time compute scaling yields adversarial robustness gains.
• Rigorously validates the rich-get-richer dynamic across model families, inference compute strategies, and attack types (black-box, white-box, multimodal).
• Demonstrates that lightweight adversarial finetuning of InternVL 3.5 gpt-oss 20B's vision encoder converts a model that gains no robustness from test-time compute into one that does, producing the first adversarially robust RL-tuned reasoning VLM.
• Shows security specifications alone (even with pre-filled conforming tokens) cannot drive robustness without underlying instruction-following ability on adversarial data.
• Extends reasoning-based robustness demonstrations to open-source models and models without RL finetuning.

Results

• Base adversarial accuracy (ε=4/255, ℓ∞): LLaVA-v1.5: 1.0% avg; FARE-LLaVA-v1.5: 20.1% avg; Delta2-LLaVA-v1.5: 61.5% avg across COCO, Flickr30k, VQAv2, TextVQA.
• Security specification experiment (Table 2): Adding explicit spec to LLaVA-v1.5 degrades attacker loss (attack easier); FARE gains a neutral/small benefit; Delta2 jumps from attacker loss 13.5 → 21.2 at step 100 and 12.4 → 21.1 at step 300.
• Inference compute scaling: Only Delta2-LLaVA-v1.5 shows a clear increase in PGD steps needed as \(K\) increases at \(\varepsilon = 64/255\); reducing \(\varepsilon\) to 16/255 allows FARE to benefit, consistent with RICH.
• InternVL 3.5 on Attack-Bard: Base model gains +0.0 adversarial accuracy from 1 to 2048 reasoning tokens; after robustifying the ViT, the model gains +4.5 accuracy points at 2048 tokens vs. +4.0 at 1024 tokens.
• Pre-filling experiment: Pre-filling responses to satisfy the security specification does not reduce attacker success in non-robust models, ruling out superficial specification satisfaction as the mechanism.

Limitations

• The rich-get-richer dynamic is fundamentally a prerequisite bottleneck: models without initial robustness gain no benefit from inference scaling, so the approach does not solve robustness for poorly-trained models.
• Naive inference scaling (prompt repetition) is tested instead of RL-based reasoning for most white-box experiments, leaving the interaction with chain-of-thought reasoning under white-box attacks underexplored.
• The paper is truncated; Section 4.3 and 4.4 full quantitative results are partially absent from the provided text.
• Compositional generalization as the underlying mechanism is hypothesized and supported indirectly; it is not directly measured or ablated.
• Experiments are concentrated on LLaVA-style models and InternVL 3.5; generalization to other VLM architectures (e.g., Flamingo-style) is not shown.

Relevance to Vision-Language Models

This work directly addresses a core open question in VLM safety: whether extended reasoning (test-time compute scaling) can defend against visual adversarial attacks, which prior work showed it largely cannot. The RICH provides a principled explanation grounded in compositional generalization and the distribution gap between clean training data and adversarially perturbed inputs — a challenge intrinsic to the visual modality of VLMs. The creation of the first adversarially robust reasoning VLM (by robustifying InternVL 3.5's vision encoder) sets a concrete new baseline for the field. For researchers tracking VLMs, this work establishes that train-time vision encoder robustness is a prerequisite for test-time reasoning defenses to function, informing both architecture and training pipeline decisions for deployable multimodal systems.