On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; systematic study of VLA robustness to multi-modal perturbations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper benchmarks mainstream Vision-Language-Action (VLA) models under 17 perturbations across four modalities (action, observation, environment, instruction) and finds that action outputs are by far the most fragile. The authors propose RobustVLA, a fine-tuning framework combining adversarial worst-case output optimization (via flow-matching) and UCB-guided input consistency regularization, achieving 12.6% absolute gains on LIBERO over the $\pi_0$ backbone and 65.6% higher real-world success with only 25 demonstrations.

Problem¶

Existing robust VLA methods (BYOVLA, GEVRM) address only visual input perturbations, rely on external LLMs causing large inference overhead, and leave action, instruction, and environment uncertainties entirely unaddressed. No prior work evaluates or defends against the full multi-modal perturbation space that deployed robots actually encounter.

Method¶

RobustVLA fine-tunes the $\pi_0$ flow-matching backbone (and optionally OpenVLA) with two complementary terms added to the base rectified-flow objective $L^{\tau}_{\pi_0}$:

Output robustness. The worst-case action perturbation $\delta$ is found by maximizing the flow-matching loss: $$\delta \in \arg\max_{\delta} \mathbb{E}\|v_\theta(\hat{A}^\tau_t, o_t) - u(\hat{A}^\tau_t | \hat{A}^1_t)\|^2$$ where $\hat{A}^1_t = A^1_t + \delta$ and $\delta$ is computed via PGD. A TRADES-style objective then minimizes both the clean loss and the worst-case loss: $$L^{\tau}_{out} = \lambda_{out} \max_\delta \mathbb{E}\|v_\theta(\hat{A}^\tau_t, o_t) - u(\hat{A}^\tau_t|\hat{A}_t)\|^2$$

Input robustness. Since semantic-preserving input perturbations should leave the optimal action unchanged, the model is trained to produce consistent actions under perturbed observations. Perturbation type selection among all 17 noise types is cast as a multi-armed bandit; the UCB algorithm selects the most harmful perturbation at each training step $n$: $$\omega^* = \arg\max_{\omega^i \in \Omega} \left[r_n(\omega^i) + \alpha\sqrt{\frac{\log n}{\omega^i(n)}}\right]$$ where the reward $r_n(\omega^i)$ is the gap between noisy and clean flow-matching losses. An additional $\ell_p$-bounded PGD noise $\eta$ is added to the selected observation. The overall loss is: $$L^{\tau}_{\text{RobustVLA}} = L^{\tau}_{\pi_0} + L^{\tau}_{in} + L^{\tau}_{out}$$

Key Contributions¶

First systematic evaluation of VLA robustness across 17 perturbation types in four modalities (action, observation, environment, instruction) using LIBERO; reveals action as the most fragile modality and identifies $\pi_0$'s diffusion head as robustness-favorable.
RobustVLA: offline adversarial fine-tuning against worst-case action noise via flow-matching maximization, interpretable as simultaneous adversarial training, label smoothing, and outlier penalization.
UCB-based automatic selection of the most harmful input perturbation during training, removing manual weight-tuning for 17 noise types.
Demonstrated generalization to OpenVLA (autoregressive backbone) with adapted token-space perturbation.

Results¶

LIBERO (17 perturbations, $\pi_0$ backbone): RobustVLA +12.6% absolute success rate over $\pi_0$ baseline; 50.6× faster inference than BYOVLA (which requires external LLMs).
LIBERO (OpenVLA backbone): +10.4% absolute gain across all 17 perturbations.
Mixed perturbation setting: +10.4% gain over baseline.
Real-world FR5 robot (4 multimodal perturbation types):
25 demonstrations: RobustVLA outperforms $\pi_0$ by 65.6% success rate.
100 demonstrations: still outperforms $\pi_0$ by 30% success rate (where $\pi_0$ saturates).
Evaluation finding: BYOVLA improves visual Gaussian by 7.3% and dead pixels by 22.3%, but average visual gain is only 4.0% and non-visual gain is +0.0%.
Backbone comparison (no robustness training): $\pi_0$ outperforms OpenVLA by 27.9% and $\pi_0$-FAST by 5.1% across perturbations.

Limitations¶

Evaluation confined to LIBERO simulation and one real-world robot platform (FR5); breadth of tasks and embodiments is limited.
Output robustness via PGD-computed worst-case noise adds training compute overhead; per-step UCB bandit adds bookkeeping but no explicit analysis of compute cost.
The POMDP robustness formulation assumes perturbations are bounded and i.i.d. at each step; correlated or adaptive adversarial perturbations are not studied.
Instruction perturbations (lexical, syntactic, adversarial prompts) are included in evaluation but the input robustness formulation focuses on observation space; how well the UCB bandit actually prioritizes instruction noise is not separately analyzed.
GEVRM baseline could not be reproduced due to missing public code, leaving one prior method uncompared.

Relevance to Vision-Language Models¶

VLA models directly extend VLMs with action generation heads (autoregressive token or diffusion-based), making this work directly relevant to practitioners deploying VLMs for embodied tasks. The finding that visual robustness methods trained on image corruption do not transfer to instruction or action modalities highlights a fundamental gap in how robustness is currently studied for multimodal models. The UCB bandit mechanism for automatically identifying harmful input perturbations is a broadly applicable training strategy for any multimodal model trained with a contrastive or generative objective. The paper's empirical result that diffusion-based action heads ($\pi_0$) are substantially more robust than autoregressive token heads ($\pi_0$-FAST) provides actionable guidance on architectural choices for VLM-based control systems.