Skip to content

Robust SuperAlignment: Weak-to-Strong Robustness Generalization for Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Standard weak-to-strong generalization improves clean accuracy in VLMs but fails to transfer adversarial robustness, even when the weak teacher is itself adversarially trained. This paper proposes Adv-W2S, which fixes this by injecting adversarial examples into the alignment objective and introducing entropy-guided re-weighting plus inverse adversarial teacher refinement to elicit zero-shot robustness from large VLMs without ground-truth labels.

Problem

As VLMs surpass human capability, reliable human supervision becomes a bottleneck—motivating "superalignment" via weak-to-strong generalization. However, the vanilla paradigm transfers only clean-sample knowledge; adversarial robustness is not transferred even from a robustly fine-tuned teacher. The root cause, identified here for the first time, is the absence of adversarial examples in the alignment objective, leaving strong VLMs exposed to adversarial attacks in zero-shot settings.

Method

Adv-W2S extends the weak-to-strong generalization objective (soft-label distillation from a weak teacher CLIP to a strong student CLIP) by two modifications:

  1. Adversarial input substitution: The student is trained on adversarial examples x̂ = x+δ (generated via PGD maximizing teacher–student KL divergence), not clean inputs. This forces the student to handle worst-case distributions even without a robust teacher.

  2. Entropy-guided uncertainty re-weighting (EUR): A per-sample weight w(x) = H_T(x)/(H_T(x)+Îş_H), where Îş_H is the median teacher entropy, adaptively down-weights high-entropy (unreliable) teacher predictions, pushing the student toward self-refinement for ambiguous samples. Theorems show low-entropy predictions enlarge the vision feature norm, expanding the classification margin and hence robustness radius.

  3. Inverse adversarial refinement (IAR): Instead of using raw teacher predictions as supervision, a refined sample xĚŚ is generated by minimizing the teacher's cross-entropy (gradient descent on input), producing a high-confidence "inverse adversary." The student aligns its adversarial predictions to these refined teacher outputs. Theorem 3 shows this also tightens clean-sample alignment as a byproduct.

The combined objective is unsupervised (uses only teacher pseudolabels). At inference, the robustly aligned CLIP vision encoder can be swapped plug-and-play into larger VLM frameworks (LLaVA, OpenFlamingo).

Key Contributions

  • First systematic identification of why vanilla weak-to-strong generalization fails to transfer robustness: missing adversarial examples in the alignment objective.
  • Adv-W2S: first adversarially robust weak-to-strong generalization framework for VLMs, operating in a fully unsupervised scheme.
  • Entropy-guided per-sample re-weighting (EUR) with supporting theoretical characterization of the entropy–margin–robustness chain.
  • Inverse adversarial example generation (IAR) as refined teacher guidance, with proof that it simultaneously improves adversarial and clean alignment.
  • Plug-and-play generalization: replacing vision encoders of LLaVA 1.5 7B and OpenFlamingo 9B transfers robustness to captioning, VQA, hallucination mitigation, and CoT reasoning.

Results

  • Zero-shot classification (14 datasets, ViT-L, ε=2/255, Auto-Attack): Adv-W2S achieves average clean 68.75% / robust 48.38%, vs. best baseline TGA at 62.11% / 45.19% — approximately +5% clean and +5.8% robust.
  • Diverse backbones (ε=2/255): ResNet101: 50.14% clean / 25.93% robust vs. next-best 47.88% / 23.40%; ViT-B/16: 55.42% / 35.20% vs. 54.09% / 30.54%.
  • Larger perturbation (ViT-L, ε=4/255): 61.56% clean / 35.19% robust vs. FARE 60.26% / 32.02%.
  • Image captioning (LLaVA, COCO CIDEr): 110.8 clean / 55.3 robust vs. PMG 103.1 / 52.8 (best prior).
  • VQA (LLaVA, VQAv2): 72.9% clean / 37.3% robust vs. FARE 70.3% / 34.5%.
  • Hallucination (POPE, LLaVA): Adv-W2S avg F1 83.8 vs. FARE 80.8.
  • CoT science QA (single-choice): 69.3% vs. PMG 68.0%.
  • Ablation: Both EUR and IAR contribute; combined gains are +5.3% clean and +5.95% robust over the base adversarial W2S.
  • Supervised variant: Adding ground-truth labels recovers ImageNet in-distribution performance (80.56% clean / 64.08% robust) but reduces zero-shot performance, revealing a distribution trade-off.

Limitations

  • In-distribution (ImageNet) clean accuracy of the unsupervised variant (75.84%) lags behind supervised adversarial fine-tuning methods (e.g., TeCoA 80.00%), reflecting a known unsupervised generalization trade-off.
  • Experiments are conducted on CLIP-family vision encoders; generalization of the framework to non-CLIP VLMs with different architectures is not directly validated.
  • The weak teacher is constrained to be smaller CLIP variants (ViT-B/32, ResNet50); whether a radically different architecture as teacher breaks the framework is unexplored.
  • IAR generates additional adversarial examples per training step, increasing computational cost relative to standard adversarial fine-tuning; cost analysis is deferred to appendices.

Relevance to Vision-Language Models

This work directly addresses a critical robustness gap that emerges as VLMs approach or exceed human performance and human supervision degrades in quality—a scalability problem central to future foundation VLM deployment. By demonstrating that adversarial robustness can be elicited through a weak-to-strong alignment paradigm rather than expensive supervised adversarial training from scratch, it offers a practically scalable path to robust VLMs. The plug-and-play encoder substitution into LLaVA and OpenFlamingo shows the technique generalizes across the VLM ecosystem, connecting adversarial robustness research to the broader multimodal alignment and superalignment agendas.