GeoDetect: Geometric Adversarial Detection for VLPs¶

🕒 Published (v1): 2026-07-16 09:02 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Adversarial detection for multimodal VLPs; ECCV 2026; geometric approach is novel angle

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

GeoDetect is a model-agnostic adversarial-example (AE) detection framework for vision-language pre-trained models (VLPs) that exploits the anisotropic geometry of their embedding spaces. The core insight—proved theoretically—is that adversarial perturbations push representations off the clean data manifold, increasing their expected distance to randomly sampled clean embeddings. Four classical geometric metrics (LID, $k$-NN, Mahalanobis distance, KDE) operationalize this signal without requiring model fine-tuning.

Problem¶

Existing adversarial-detection methods were designed for unimodal classifiers trained with cross-entropy loss. VLPs use contrastive objectives that produce structurally anisotropic, jointly-aligned embedding spaces, so the assumption that detection signals transfer from unimodal models is unverified. The only prior multimodal detector (PIP) is restricted to VQA architectures with explicit cross-attention and cannot generalize to retrieval or zero-shot classification tasks.

Method¶

GeoDetect analyzes the geometry of VLP encoder representations at inference time via a three-stage pipeline:

Theoretical grounding. The authors formally establish that VLP embeddings are anisotropic (verified via isotropy measures $I_1$, $I_2$ and normalized effective rank $\widehat{\text{ER}}$). Two assumptions—anisotropic covariance ($\Sigma \neq cI$) and manifold proximity of clean samples—lead to Lemmas 1–2 and Theorem 1, which proves the expected-distance gap: $\mathbb{E}_{z_u \sim p(\cdot)}[\|z_i' - z_u\|] > \mathbb{E}_{z_u \sim p(\cdot)}[\|z_i - z_u\|]$ for adversarial $z_i'$ satisfying $\|z_i' - \mathcal{M}\| \geq \gamma > 3\alpha$.
Score extraction. For a query embedding $z_i$ and a minibatch of clean reference embeddings $\{z_j\}_{j=1}^n$, four metrics are computed: $$\text{Metric}(\cdot,\cdot) = \begin{cases} k\text{-NN}(z_i,\{z_j\}) = \frac{1}{k}\sum_{j=1}^k \|z_i - z_j\|_2 \\ \widehat{\text{LID}}(z_i,\{z_j\}) = \left(-\frac{1}{k}\sum_{j=1}^k \log\frac{r_j}{r_{\max}}\right)^{-1} \\ \text{KDE}(z_i,\{z_j\};H) = \frac{1}{n}\sum_{j=1}^n K_H(z_i-z_j) \\ \text{Mahal}(z_i) = \sqrt{(z_i-\mu)^\top \Sigma^{-1}(z_i-\mu)} \end{cases}$$ LID is extracted layer-wise across the full encoder stack; for fused VLPs the multimodal encoder layers are additionally included. The other three metrics operate only on the image encoder output.
Detection. For $k$-NN, Mahalanobis, and KDE, a threshold $\tau$ defines $\mathcal{H}(s_i) = \mathbb{I}(s_i > \tau)$. For LID, multi-layer features feed a logistic regression classifier.

Key Contributions¶

Formal characterization of anisotropy across a broader family of VLPs (beyond CLIP), measured via $I_1$, $I_2$, and $\widehat{\text{ER}}$.
Theoretical proof (Theorem 1) that adversarial embeddings have strictly larger expected distances to clean reference points under the anisotropic manifold setting.
GeoDetect: a training-free, fine-tuning-free, model-agnostic detection framework applicable to both fused (ALBEF, TCL) and aligned (CLIP) VLPs and to both unimodal and multimodal attacks.
Empirical verification of off-manifold deviation via PCA energy analysis, low-rank reconstruction error, and per-dimension KL divergence.

Results¶

The paper text provided is truncated before the quantitative results tables. From what is present: - GeoDetect is reported to achieve "consistently high AUC scores" across diverse VLP architectures and attack types, including adaptive attacks. - The method is evaluated on zero-shot classification and image-text retrieval tasks; the baseline MCM (softmax-normalized similarity, CLIP-only, classification-only) is incompatible with retrieval settings, limiting direct comparison there. - Empirical evidence in Figs. 3a–c confirms that adversarial embeddings concentrate energy in low-singular-value directions, exhibit higher PCA reconstruction residuals (top $K=10$ clean PCs), and show larger KL divergence along leading PCA axes (top 1–20 components).

(Full benchmark numbers were not included in the provided text.)

Limitations¶

Full numerical results, architecture-level AUC breakdowns, and comparisons against additional baselines are in sections not provided (text is truncated before the results tables).
Theoretical guarantees require the condition $\gamma > 3\alpha$; whether this holds uniformly across all VLP variants and attack budgets is not verified beyond CLIP and ALBEF.
The manifold-Gaussian approximation (Assumption 2) is a local linearization; its accuracy may degrade for large perturbation radii or highly non-linear manifolds.
MCM is the only detection baseline compared against; the absence of stronger baselines (e.g., energy-based scores, PIP on VQA) limits the scope of the comparison reported so far.
No investigation of text-only adversarial perturbations ($Da = \{(x_i, t_i')\}$); the framework targets image-perturbed or both-perturbed inputs.

Relevance to Vision-Language Models¶

GeoDetect directly targets a known safety gap in VLPs: contrastive pre-training reshapes embedding geometry in ways that invalidate unimodal detection assumptions, and this work is the first to provide a theory-grounded, architecture-agnostic remedy. The finding that anisotropy is a universal property of VLP embedding spaces (CLIP, ALBEF, TCL, and others) has implications beyond detection—it suggests that geometric analysis is a productive lens for understanding how multimodal alignment distorts representation space under distributional shift. For researchers building on CLIP or similar dual-encoder architectures, GeoDetect's lightweight, no-fine-tuning design offers a practical plug-in safety layer compatible with both retrieval and classification pipelines.