Immuno-VLM: Immunizing Large Vision-Language Models via Generative Semantic Antibodies for Open-World Trustworthiness

🕒 Published (v1): 2026-05-29 02:22 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; immunizing VLMs against open-world adversarial semantic vulnerabilities

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Immuno-VLM addresses the "Hubris of Semantics" in large VLMs — their tendency to force-fit OOD inputs into known categories with high confidence — by borrowing the biological principle of thymic negative selection. The framework uses an LLM as a "Computational Thymus" to hallucinate textual descriptions of near-OOD concepts ("Semantic Antibodies"), projects them into the shared embedding space to bound decision regions, and trains a lightweight adapter to explicitly repel these antibodies while retaining in-distribution performance.

Problem

CLIP-style VLMs trained via contrastive alignment develop a retrieval bias: cosine similarity maximization over a dense semantic manifold causes any visual input, including OOD anomalies, to map to the nearest known class with high confidence. This "Open-World Trustworthiness Paradox" is exacerbated in high-dimensional spaces (\(d \geq 512\)) where classical OOD defenses (MSP, Energy Scores, AIS random detectors) fail due to the concentration-of-measure phenomenon — randomly sampled negatives are asymptotically orthogonal to all class prototypes and contribute zero boundary information.

Method

The pipeline has four stages:

1. Antigen Profiling: For each known class \(k\), the class prototype \(\mu_k\) is estimated by solving a constrained optimization on the unit hypersphere \(S^{d-1}\), fitting a Von Mises-Fisher distribution \(\text{vMF}(\mu_k, \kappa_k)\) while regularizing toward the text embedding \(t_k = \phi_t(\text{name}_k)\) to close the modality gap (Theorem 3.4: the optimal \(\mu_k^*\) lies on the geodesic between the visual mean \(\bar{v}_k\) and \(t_k\)).

2. Semantic Antibody Generation: An LLM is prompted per class to generate two types of antibodies: (a) Hard Semantic Negatives — visually similar concepts from disjoint categories (e.g., "wolf", "robot dog" for class "dog"), and (b) Contextual Anomalies — the known object in physically impossible contexts (e.g., "car underwater"). Generated candidates are filtered by a band-pass similarity condition \(\delta_\text{safe} < \langle \phi_t(a), \mu_k \rangle < \delta_\text{risk}\) to avoid synonyms (auto-immune toxicity) and orthogonal irrelevants.

3. Vaccination: A lightweight residual adapter \(f_\theta: \mathcal{Z} \to \mathcal{Z}\) is trained (frozen encoders) via a Push-Pull loss: $\(\mathcal{L}_\text{vac} = \mathcal{L}_\text{pull} + \lambda \mathcal{L}_\text{push} + \eta\|\theta\|_2^2\)$ where \(\mathcal{L}_\text{pull}\) is a vMF softmax loss over known classes and \(\mathcal{L}_\text{push}\) is a margin hinge loss \(\sum_{a \in A_y} \max(0, \cos(f_\theta(\phi_v(x)), \phi_t(a)) - m)^2\) that enforces a "sterile moat" of width \(m\) between visual embeddings and antibody directions.

4. Inference: An OOD score \(\text{DIS} = s_+ - \lambda_\text{inf} s_-\) is computed, where \(s_+ = \max_k \langle z, \mu_k \rangle\) and \(s_- = \max_{a \in A} \langle z, \phi_t(a) \rangle\), with rejection at an EVT-calibrated threshold \(\tau_\text{evt}\).

The key theoretical result (Theorem 3.3: Antibody Covering Bound) shows that if antibody set \(A_\delta\) forms a \(\delta\)-cover of the semantic boundary and the adapter enforces margin \(m > \epsilon_\text{align} + \delta\), the FPR on open-world samples in the \(\gamma\)-neighborhood is bounded by the VLM alignment error \(\epsilon_\text{align}\). Theorem 3.5 proves that random AIS negatives in \(\mathbb{R}^{512}\) satisfy \(P(|\langle z_\text{neg}, \mu_k \rangle| \geq \epsilon) \leq 2\exp(-d\epsilon^2/2)\), formalizing why random pixel-space or hyperspherical sampling is infeasible.

Key Contributions

• Generative Semantic Antibodies: First use of an LLM as a Computational Thymus to actively hallucinate near-OOD textual descriptors that densely cover the decision boundary in semantic space, bypassing pixel-space GAN scaling failures.
• Antibody Covering Bound (Theorem 3.3): Rigorous proof that semantic boundary coverage by a finite antibody set (\(M \approx 100\)) suffices to bound open-space FPR — exponentially more efficient than isotropic random sampling (Theorem 3.5).
• Text-Regularized vMF Prototype Estimation (Theorem 3.4): Closed-form solution showing the optimal class prototype lies on the geodesic arc between the visual centroid and text embedding, correcting the modality gap.
• Push-Pull Vaccination Loss with EVT Inference: Lightweight adapter training scheme that immunizes the visual encoder against antibody directions while preserving closed-set accuracy; EVT-calibrated rejection threshold requires no held-out OOD data.
• Open Space Generalization Bound (Theorem 3.7): Formal guarantee that minimizing empirical risk on synthetic antibodies transfers to low open-space risk on unseen real anomalies, via covering number arguments.

Results

• OOD detection on ImageNet-1K + four OOD benchmarks: Immuno-VLM achieves state-of-the-art, improving detection of adversarial semantic shifts by >16% over standard zero-shot CLIP baselines while preserving in-distribution recognition accuracy.
• Compared to passive discriminative baselines (MSP, Energy Scores, Activation Shaping) and text-modality methods (MCM), Immuno-VLM is the only proactive approach that explicitly bounds the open space.
• \(M \approx 100\) antibodies per class suffice for effective semantic boundary coverage where random \(\mathbb{R}^{512}\) sampling would require exponentially many points.

(Note: full per-benchmark AUROC/FPR95 tables were not included in the provided excerpt.)

Limitations

• LLM dependency: Antibody quality is bounded by the reasoning capability and imagination of the underlying LLM (GPT-4 in this work); poor or hallucinated antibodies can degrade coverage.
• Theoretical alignment error floor: The FPR bound in Theorem 3.3 is \(\epsilon_\text{align}\), the inherent VLM modality gap — this cannot be reduced by the method itself.
• Fixed antibody set: Antibodies are generated offline per class; dynamic or streaming classes require re-generation.
• Semantic filtration thresholds \(\delta_\text{safe}, \delta_\text{risk}\): The band-pass filter is manually tuned; suboptimal values risk auto-immune toxicity or orthogonality failure.
• Text-only antibodies: Contextual anomalies are described textually; visually idiosyncratic OOD samples not well-described in language may evade the boundary.

Relevance to Vision-Language Models

This work directly targets a structural vulnerability of contrastive VLMs (CLIP, ALIGN) in open-world deployment, where semantic alignment training creates a closed-world retrieval bias. It reframes OOD detection from a passive post-hoc scoring problem into an active representation-shaping problem, using the generative capacity of LLMs to construct explicit negative knowledge — a complementary axis to standard VLM alignment research. The Antibody Covering Bound and concentration-of-measure analysis provide rare rigorous theory explaining why high-dimensional VLM latent spaces resist classical OOD methods, which has direct implications for safety-critical VLM deployment (medical imaging, autonomous driving). The lightweight adapter paradigm (frozen encoders + \(f_\theta\)) is also directly applicable to any CLIP-backbone system without retraining.