Skip to content

Hierarchically Robust Zero-shot Vision-language Models

🕒 Published (v1): 2026-04-20 21:42 UTC · Source: Arxiv · Venue: CVPR 26 · link

Why this paper was selected

Hierarchically robust VLM fine-tuning preserves zero-shot under adversarial attack

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing adversarial fine-tuning for CLIP focuses solely on base (leaf) classes, leaving models vulnerable to adversaries crafted at superclass levels and failing to generalize robustness up the label hierarchy. This paper proposes a hierarchical adversarial fine-tuning framework that embeds class and image hierarchies in hyperbolic (Poincaré ball) space, generating adversarial perturbations across multiple levels of abstraction simultaneously. On 15 zero-shot datasets it surpasses prior state-of-the-art on both clean and robust accuracy.

Problem

Standard adversarial fine-tuning of VLMs (TeCoA, FARE, PMG) aligns adversarial image embeddings against flat, base-class text embeddings only. This creates two compounding failures: (1) robustness does not generalize to superclasses—adversarial samples generated at the superclass level transfer effectively to defeat base-class classifiers, and (2) the Euclidean classifier's viable margin is bounded in \((0,1)\), limiting the generality of the learned perturbations. No prior method exploits the inherent tree structure of label spaces to produce more universal adversaries.

Method

The framework represents the class hierarchy (WordNet for ImageNet; ChatGPT-generated for other datasets) as a Poincaré ball embedding. CLIP's Euclidean embeddings are lifted into hyperbolic space via the exponential map \(\text{exp}^r_0(\cdot)\); leaf-level nodes lie near the Poincaré boundary (high \(\ell_2\) norm) while root-level nodes lie near the origin (low norm).

Hierarchical image embedding: Since no "universal superclass image" exists, parent-level image embeddings are synthesized by Hyperbolic Averaging (HypAvg; Einstein midpoint) over child embeddings in the mini-batch, recursively up to root level \(L\).

Hierarchy-preserving Image-Text Alignment (HITA): The adversarial objective sums weighted cross-entropy losses across all \(L+1\) hierarchy levels: $\(\mathcal{L} = \max_{\|\delta_X\|_\infty\leq\epsilon_X,\,\|\delta_T\|_\infty\leq\epsilon_T} \sum_{l=0}^{L} \omega_l \mathcal{L}_{\text{CE}}\!\left(p^l(x{+}\delta_X),\,y(c)\right), \quad \omega_l = 1 - \tfrac{l}{L+1}\)$ where both image (\(\delta_X\)) and text (\(\delta_T\)) perturbations are optimized jointly via PGD.

Hierarchy-aware Negative Set Augmentation (HNSA): The probability for class \(c\) at level \(l\) is augmented with negative classes at levels \(l{-}1\) and \(l{+}1\) (non-adjacent classes), sharpening hierarchical discrimination.

Norm-ordering constraints: \(\mathcal{L}^\text{Label}_\text{gap}\) enforces that \(\|\psi(\hat{t}^{l+1}_c)\|_2 < \|\psi(\hat{t}^l_c)\|_2\), preserving parent-before-child ordering. \(\mathcal{L}^\text{Intra}_\text{gap}\) and \(\mathcal{L}_\text{vic}\) maintain the spread of individual image embeddings around their base-class text anchor.

Theoretical margin: The log margin in hyperbolic space, \(m^r_{c,c'}(\eta)\), grows to \(\infty\) as \(\eta\to 1/\sqrt{r}\), while the Euclidean counterpart \(m^\text{Euc}(\eta) = 2\lambda\eta^2\) stays bounded. Multiple hierarchy levels realize multiple distinct margin sizes in \((0,\infty)\), producing more universal adversarial perturbations.

A multi-tree (hierarchical forest) variant aggregates losses over 2–5 independently constructed trees to increase semantic variety.

Key Contributions

  • Reformulation of adversarial fine-tuning as a hierarchical scheme over PoincarĂ©-ball embeddings, enabling adversaries that generalize across the full class hierarchy rather than fixating on leaf classes.
  • HypAvg-based construction of parent-level image embeddings from mini-batch children, avoiding the need for explicit superclass images.
  • HITA loss combining image and text adversarial perturbations across all hierarchy levels, with HNSA negative augmentation.
  • Formal theorem (Theorem 1) quantifying the \((0,\infty)\) margin range in hyperbolic vs. bounded range in Euclidean classifiers and its connection to adversarial generalization.
  • Evaluation across 15 datasets, plus adversarially robust retrieval, captioning, and medical imaging tasks.

Results

  • Clean accuracy (Table 1, avg. 15 datasets): Ours 62.14% vs. AoS 61.70%, FARE 59.67%, PMG 57.36%, TeCoA 52.62%; 5-tree variant 62.49%.
  • PGD-20 robust accuracy (Table 2, \(\epsilon=1/255\), avg. 15 datasets): Ours 44.34% vs. AoS 43.88%, FARE 37.93%; 5-tree variant 45.39%; mean gain of +6.4% over FARE.
  • Across perturbation radii (Table 3): At \(2/255\): 27.19% vs. AoS 26.70%; at \(4/255\): 9.92% vs. AoS 9.51%.
  • Multiple attacks (Table 4, \(\epsilon=2/255\)): Ours PGD 31.64%, CW 30.55%, AA 29.49% vs. AoS PGD 30.48%, CW 29.45%, AA 28.70%.
  • Architecture generalization (Table 5): Consistent gains over baselines on ViT-B, ViT-L, and ResNet-50.
  • With 5 trees, surpasses AoS by +1.5% robust accuracy despite AoS using \(10\times\) more image and \(50\times\) more text augmentations.

Limitations

  • Requires a hierarchical taxonomy per dataset; for datasets without predefined ontologies (like WordNet), ChatGPT-generated hierarchies need manual review, introducing a human bottleneck.
  • Hierarchy depth is fixed at \(L=5\) for ImageNet; optimal depth for other domains is not systematically studied.
  • Hyperbolic projections and recursive HypAvg computations add training overhead not explicitly quantified.
  • Text-level adversarial perturbations (\(\delta_T\)) operate in continuous embedding space, departing from realistic text-level threat models.
  • Evaluation is limited to \(\ell_\infty\)-norm attacks; robustness to \(\ell_2\) or semantic attacks is not reported.

Relevance to Vision-Language Models

This work directly addresses a known weakness of robustly fine-tuned CLIP variants: their failure to withstand adversaries crafted at any level of the label hierarchy, not just leaf classes. By grounding the alignment objective in hyperbolic geometry, it exploits the hierarchical semantic structure already implicit in CLIP's text encoder (via natural language superclasses), pointing toward a general principle that multi-granularity label spaces should be reflected in the optimization geometry. The theoretical connection between Poincaré-ball embedding depth, margin size, and adversarial generalizability provides a principled basis for future work combining hyperbolic representations with other VLM fine-tuning paradigms (prompt tuning, LoRA, etc.). Results on retrieval and captioning tasks extend the robustness gains beyond classification, increasing relevance to broader VLM deployment scenarios.