A Provable Energy-Guided Test-Time Defense Boosting Adversarial Robustness of Large Vision-Language Models¶

🕒 Published (v1): 2026-03-27 20:53 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

CVPR; provable energy-guided test-time defense for VLM adversarial robustness

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ET3 (Energy-Guided Test-Time Transformation) is a training-free, plug-and-play defense that enhances adversarial robustness of large vision-language models (LVLMs) by minimizing the logit-derived energy of adversarial inputs at inference time via gradient descent through the CLIP visual encoder. The method is backed by a formal proof showing that under local linearity and gradient-norm-ratio assumptions, the transformation provably yields correct classification. ET3 improves robustness for zero-shot classification (CLIP), image captioning, and VQA without modifying any model weights.

Problem¶

LVLMs such as LLaVA rely on CLIP as their visual encoder, which is the primary attack surface for adversarial perturbations. Existing defenses are costly: adversarial training (AT) is expensive and leaves a clean/perturbed accuracy gap; adversarial purification requires auxiliary generative models; test-time prompt tuning (TPT, R-TPT) incurs substantial inference overhead from multiple augmented views; and none of these approaches come with a theoretical guarantee of success on the transformed input.

Method¶

ET3 interprets a standard softmax classifier through the Energy-Based Model (EBM) lens. The energy at input \(\mathbf{x}\) is the negative LogSumExp of logits:

\[E(\mathbf{x}) = -\log\Bigl(\sum_{k=1}^{K} \exp\bigl(f_\theta(\mathbf{x})_k\bigr)\Bigr)\]

Given a test image \(\mathbf{x}\) and a defense radius \(\epsilon\), ET3 solves

\[\tilde{\mathbf{x}} = \arg\min_{\mathbf{x}^* \in \mathcal{B}_\epsilon(\mathbf{x})} E(\mathbf{x}^*)\]

via projected gradient descent for \(T\) steps:

\[\mathbf{x}^{(t)} = \Pi_{\mathcal{B}_\epsilon(\mathbf{x})}\!\Bigl(\mathbf{x}^{(t-1)} - \alpha \nabla_{\mathbf{x}} E\bigl(\mathbf{x}^{(t-1)}\bigr)\Bigr)\]

with \(\ell_2\)-ball projection. The gradient of the energy decomposes as \(\nabla_\mathbf{x} E = -\sum_k e_k \mathbf{g}_k\) (softmax weights times logit-gradient vectors), so minimizing energy shifts probability mass toward higher-logit classes.

For zero-shot CLIP, logits are image-text cosine similarities computed against ImageNet-21K proxy labels, enabling energy minimization without ground-truth labels. For LVLMs (LLaVA), ET3 runs only through the frozen CLIP vision encoder; the refined visual embedding is then passed to LLaVA's projection and language generation layers unchanged — the LLM is never involved in the optimization.

Key Contributions¶

A training-free, lightweight test-time defense (ET3) requiring only \(T \approx 1\)–few gradient steps through a frozen visual encoder.
A formal proof (Theorem 4.1) that under local linearity in \(\mathcal{B}_\epsilon\) and a gradient-norm-ratio condition \(C \|e_{\hat{y}} \mathbf{g}_{\hat{y}}\| < \|e_y \mathbf{g}_y\|\), ET3 provably produces a correctly classified output for both clean and adversarial inputs.
Extension to zero-shot CLIP (using large ImageNet-21K label sets as energy anchors) and to generative LVLM tasks (captioning, VQA) without any model fine-tuning.
Demonstrated compatibility with existing robust encoders (TeCoA, FARE/Robust CLIP) as a composable plug-in that further lifts their robustness.
Verified resistance to adaptive attacks.

Results¶

Zero-shot classification (CLIP + TeCoA/FARE): ET3 consistently improves robust accuracy across 14 of 15 evaluated benchmark datasets under AutoAttack (APGD-CE + APGD-DLR, 100 iterations, \(\epsilon_a = 4/255\)), including for models trained at the weaker \(\epsilon_t = 2/255\).
LLaVA image captioning and VQA: Qualitative examples show ET3 recovers correct captions and answers (e.g., "A man is surfing on a wave in the ocean" / correct team "Reds") that both the undefended and standard Robust CLIP backbones fail on under attack.
Single optimization step through the CLIP encoder suffices for LVLM tasks, with negligible inference overhead.
(Full quantitative tables truncated in the provided text.)

Limitations¶

Theoretical guarantee is proven only for binary classifiers under local linearity and a gradient-norm-ratio assumption; extension to the multi-class and LVLM regime is empirical.
Defense radius \(\epsilon\) and step size \(\alpha\) are hyperparameters that must be tuned; the text does not report sensitivity analysis in the provided excerpt.
Energy minimization is performed through the CLIP encoder only; perturbations targeting the LLM backbone or the multimodal projection layer are not addressed.
Effectiveness depends on the quality of the underlying robust encoder — "weaker" models trained at smaller \(\epsilon_t\) show improvements but from a lower baseline.
No quantitative VQA or captioning metrics (CIDEr, BLEU, accuracy) are reported in the provided text excerpt.

Relevance to Vision-Language Models¶

ET3 directly targets the CLIP visual encoder bottleneck shared by most production LVLMs (LLaVA, OpenFlamingo, Qwen-VL), making it broadly applicable without retraining any component of those systems. It demonstrates that the EBM reinterpretation of softmax logits — previously explored for classifiers — transfers cleanly to the CLIP similarity-scoring regime and propagates robustness improvements through the full multimodal pipeline. The result is a practical, composable defense layer for LVLM deployment that is orthogonal to and stackable with adversarial training of the encoder. The formal proof, though restricted to binary classifiers, provides a theoretical anchor for energy-minimization as a principled robustness mechanism rather than a heuristic.