Semantic Robustness Certification for Vision-Language Models¶

🕒 Published (v1): 2026-06-17 09:15 UTC · Source: Arxiv · Venue: ICML · link

Why this paper was selected

ICML; first certified robustness framework for semantic distribution shifts in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces the first framework for certifying VLM robustness under semantic-level transformations (e.g., shape, style, scene) by using text prompt pairs as semantic proxies to construct parameterized embedding-space transformations. By exploiting the cosine-similarity geometry of the shared VLM embedding space, the framework derives closed-form decision boundaries and produces precise prediction-invariant intervals over a semantic extent $\phi$, requiring no additional data per variation.

Problem¶

Existing robustness certification methods address pixel-level $L_p$ perturbations or hand-crafted geometric transforms, neither of which captures real-world semantic variation (e.g., "triangular" vs. "round" shape). Methods that do handle semantic variation (ApproxLine, GCERT) require training generative models with substantial in-domain data for each new semantic, making them impractical. VLMs are known to be fragile to such distributional shifts, yet no data-free semantic certification framework existed.

Method¶

The framework operates in the VLM's shared unit embedding space $\mathcal{S}^{d-1}$:

Semantic plane construction. Given a prompt pair $(t_a, t_{a'})$ representing source and target semantics, compute textual embeddings $u_a, u_{a'} \in \mathcal{S}^{d-1}$ and define the 2D semantic plane $\mathcal{P}_{a,a'} = \text{span}\{u_a, u_{a'}\}$. Semantic strengths are additive under linear decomposition (Remark 4.2), so variation confined to $\mathcal{P}_{a,a'}$ fully captures the intended semantic shift.
Semantic transformation. Decompose the image embedding as $z = z_\parallel + z_\perp$ where $z_\parallel \in \mathcal{P}_{a,a'}$ and $z_\perp \perp \mathcal{P}_{a,a'}$. Establish an orthonormal basis $(e_1, e_2)$ for $\mathcal{P}_{a,a'}$ via Gram–Schmidt on $(u_a, u_{a'})$. Define the parameterized transformation $$\gamma(\phi) := r(\cos\phi\, e_1 + \sin\phi\, e_2) + z_\perp, \quad \phi \in [\phi_a, \phi_{a'}],$$ where $r = \|z_\parallel\|_2$ and extent $\phi$ controls in-plane semantic strength. $z_\perp$ is preserved so semantics unrelated to $(a, a')$ remain unchanged.
Certification. The VLM classifier induces Voronoi decision regions on $\mathcal{S}^{d-1}$ with pairwise bisectors $\mathcal{B}_{c,c'} = \{e : \langle e, u_c - u_{c'}\rangle = 0\}$. Substituting $\gamma(\phi)$ yields the closed-form pairwise margin $$m_{c,c'}(\phi) = A_{c,c'}\cos\phi + B_{c,c'}\sin\phi + C_{c,c'},$$ whose zeros give all decision-boundary crossings analytically. Sorting these yields a complete labeled partition $\mathcal{S} = \{((\phi_\ell, \phi_{\ell+1}), y_\ell)\}$ of $[\phi_a, \phi_{a'}]$ into prediction-invariant subintervals.
Misalignment robustness. For cross-modal embedding mismatch bounded by $\delta$, Proposition 4.6 provides a stability condition: $m_{\hat{y},c'}(\phi) > \delta\|u_{\hat{y}} - u_{c'}\|_2$ for all $c' \ne \hat{y}$ guarantees $f(e) = \hat{y}$ for all $e$ in the $\delta$-neighborhood $\Gamma_\delta(\phi)$.

The target extent $\phi_{a'}$ is specified either via the target text embedding (text-specified setting) or via a reference image exhibiting the target semantic (image-specified setting).

Key Contributions¶

First semantic robustness certification framework for VLMs that requires no additional training data per variation.
Text-prompt-based semantic proxies that leverage open-vocabulary VLM grounding to specify arbitrary semantic variations within the shared embedding geometry.
Closed-form characterization of the VLM Voronoi decision boundary enabling exact, analytic computation of all prediction-invariant intervals over the semantic extent.
Misalignment-aware certificate bounds parameterized by a cross-modal mismatch budget $\delta$, localizing prediction-change uncertainty to $\mathcal{U}_{c,c'}(\delta)$.

Results¶

Evaluated on CLIP VLMs using both synthetically generated images and real-world data across diverse semantic variation types (shape, style, scene, etc.).
Certified prediction-invariant intervals are visualized and shown to match actual prediction changes along the semantic extent $\phi \in [0,1]$ (e.g., "Gyoza" prediction holds for $\phi < 0.77$, flips to "Samosa" beyond it).
Compared against ExactLine (the only complete certifier applicable without auxiliary inputs in this setting); the paper demonstrates qualitative coverage across multiple attribute types (shape, size, color, texture, scene).
(Quantitative aggregate benchmark numbers are not available in the provided excerpt.)

Limitations¶

Assumption 4.1 (cosine similarity as the faithful measure of semantic strength) may not hold precisely for all semantic concepts, potentially loosening certificates in practice.
The quality of the semantic plane $\mathcal{P}_{a,a'}$ depends critically on prompt pairs sharing the same template and content words; prompts differing beyond the intended attribute token can leak unintended semantic factors into the plane.
Framework is developed and tested specifically for dual-encoder contrastive VLMs (CLIP family); generalization to generative or autoregressive VLMs is not addressed.
The cross-modal misalignment bound $\delta$ must be supplied as a prior assumption; empirically estimating it for a given VLM and semantic domain is non-trivial.

Relevance to Vision-Language Models¶

This work directly exploits the defining architectural property of contrastive VLMs—alignment of image and text in a shared cosine-similarity geometry—as the mathematical substrate for formal certification, making it a VLM-specific contribution rather than a generic neural network verification technique. It addresses a well-known but poorly-formalized fragility of CLIP-style models: sensitivity to semantic distribution shifts that neither adversarial training studies nor pixel-level certificates capture. For researchers tracking VLMs, this provides the first data-free tool to characterize where in semantic concept-space a model's predictions are stable, with direct implications for deployment in safety-critical settings. The text-prompt interface also connects naturally to ongoing work on VLM prompt engineering and concept-level interpretability.