Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?¶

🕒 Published (v1): 2026-06-30 09:35 UTC · Source: Arxiv · link

Why this paper was selected

Visual ambiguity detection for VLMs; essential for reliable deployed systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Existing uncertainty estimators for VLMs fail on visually ambiguous inputs because overconfident visual embeddings suppress output diversity under stochastic decoding, and text-based perturbations introduce prompt sensitivity rather than probing visual ambiguity. This paper proposes Visual Semantic Entropy (VSE), which perturbs only the input image at a fixed small scale, clusters generated answers into semantic prototypes, and quantifies uncertainty as mass-weighted pairwise dispersion among those prototypes. VSE achieves state-of-the-art uncertainty estimation across 5 VLMs and 5 VQA benchmarks, particularly on visually adversarial inputs.

Problem¶

Three failure modes plague current VLM uncertainty estimators: (1) Semantic Entropy (SE) relies on decoding stochasticity, but overconfident visual embeddings produce sharply peaked conditional answer distributions, yielding near-zero SE even for incorrect predictions on ambiguous images. (2) Distance-based methods (SNNE, KLE) aggregate pairwise embedding distances over sampled text outputs, inflating uncertainty due to wording variation among semantically identical answers. (3) Methods using textual paraphrasing or joint image-text perturbations (C&U, VL-Uncertainty) produce output diversity dominated by prompt sensitivity rather than visual content — text purity $P_T$ consistently exceeds image purity $P_I$ (e.g., 51.9% vs 36.4% on the Easy split for Qwen2.5-VL), and cosine distance from text perturbations (mean $0.056 \pm 0.014$) greatly exceeds that from image perturbations (mean $0.012 \pm 0.007$) in multimodal embedding space.

Method¶

VSE has three stages operating on a fixed question $q$ and image $v$:

1. Visual Perturbation. Generate $M$ perturbed images $v_m = \mathcal{T}(v; \xi_m)$ using a fixed small perturbation scale. The operator $\mathcal{T}$ preserves high-level semantic content while introducing local pixel-space variation. The text query is never modified.

2. Answer Sampling. Feed each $(q, v_m)$ pair to the VLM to obtain $M$ answer samples $\{a_m\}$.

3. Prototype Semantic Aggregation (ProtoSem). Cluster $\{a_m\}$ into $K$ semantic groups $\{c_k\}$ by semantic distance $d(\cdot,\cdot)$. For each cluster, select the medoid prototype: $$p_k = \arg\min_{a \in c_k} \sum_{a' \in c_k} d(a, a')$$ Let $w_k = |c_k|/M$ be the cluster mass. Uncertainty is the expected distance between two independently drawn prototypes: $$\tilde{u} = \sum_{k \neq k'} w_k w_{k'} \, d(p_k, p_{k'})$$ This eliminates intra-cluster wording inflation (only inter-prototype distances count) while weighting genuine semantic disagreements by their empirical frequency.

Key Contributions¶

Formal analysis of three failure modes in VLM uncertainty estimators, supported by LogitLens-based visual entropy diagnostics and cluster purity experiments across four models
VSE: an image-only perturbation + ProtoSem pipeline that decouples visual uncertainty from prompt sensitivity and wording variation
Benchmark evaluation of multiple UE methods across 5 VLMs (Qwen2.5-VL, Gemma3, InternVL2.5, LlaVA-NeXT, and one more) and 5 VQA datasets including the visually adversarial VILP split

Results¶

VSE establishes new state-of-the-art AUC for uncertainty estimation across all 5 VLMs and 5 datasets (full tables in truncated section)
On VILP Adversarial split: after image perturbation, visually confident samples (low $H_\text{vis}$) show substantially higher SE compared to stochastic decoding alone (Fig. 3, Qwen2.5-VL), confirming that visual perturbation recovers suppressed uncertainty
Text purity $P_T > P_I$ across all four evaluated models on VILP Easy and Adversarial splits (Table 1), with gaps up to 15 percentage points (Qwen2.5-VL Easy: 51.9% vs 36.4%), supporting the text-dominance hypothesis
VSE is especially effective on visually adversarial datasets where text perturbation-based baselines degrade

Limitations¶

Computationally more expensive than decoding-only methods: requires $M$ full VLM forward passes per query
Performance depends on the design of the perturbation operator $\mathcal{T}$ and scale; too aggressive a scale may exit the local semantic neighborhood of $v$
Full quantitative results (AUC tables, ablations) are not present in the provided text excerpt
The method targets visual epistemic uncertainty in VQA; applicability to generation tasks or non-VQA settings is not addressed
Clustering requires choosing $K$ and a semantic distance function $d(\cdot,\cdot)$, whose sensitivity is not fully characterized in the available text

Relevance to Vision-Language Models¶

VSE directly addresses a blind spot in VLM deployment: models can be confidently wrong when visual evidence is ambiguous, and standard uncertainty tools inherited from NLP miss this because they probe only text-space diversity. The finding that visual embeddings can be overconfident enough to suppress decoding stochasticity is a structural insight about VLM internals (confirmed via LogitLens projections), with direct consequences for calibration and hallucination research. The text-dominance finding in joint perturbation methods challenges a growing body of robustness work that conflates prompt sensitivity with model uncertainty. For researchers tracking VLMs, VSE provides both a diagnostic lens and a practical estimator that better captures the modality-specific failure modes that distinguish VLMs from pure LLMs.