Skip to content

GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similarity

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GLSim is a training-free object hallucination detection framework for large vision-language models (LVLMs) that fuses two complementary internal embedding-similarity signals—a global scene-level score and a local spatial-grounding score—into a single weighted detector. Neither signal alone is reliable: global similarity suffers from spurious co-occurrence associations, while local similarity misfires on visually similar distractors. Their combination consistently outperforms all prior detection baselines without requiring ground-truth annotations or external judge models.

Problem

Existing object hallucination (OH) detection methods rely on one of two perspectives in isolation: global signals (NLL, SVAR) that capture overall semantic context but are fooled by plausible contextual co-occurrences, or local signals (Internal Confidence) that probe specific visual patches but are tricked by visually similar distractors. No prior method integrates both, leaving a systematic reliability gap across diverse hallucination failure modes.

Method

GLSim assigns each object mention \(o\) extracted from LVLM output a scalar score via a weighted sum of two cosine-similarity terms, all computed from internal LVLM representations—no extra model or annotation needed.

Local score (\(s_\text{local}\)): Applies a Visual Logit Lens (VLL)—multiplying each visual token's hidden state \(h_l(v_i)\) by the unembedding matrix \(W_U\) and softmaxing—to rank image patches by their probability of predicting object token \(o\). The Top-\(K\) most relevant patches are selected as \(I(o)\), and their average cosine similarity with the object token's hidden representation \(h_{l'}(o)\) yields \(s_\text{local}\).

Global score (\(s_\text{global}\)): Computes cosine similarity between \(h_{l'}(o)\) and the hidden state of the final instruction-prompt token \(h_l(v,t)\), which acts as a condensed multimodal summary encoding scene-level context.

Final score: \(s_\text{GLSim} = w \cdot s_\text{global} + (1-w) \cdot s_\text{local}\), with \(w=0.6\) found to be near-optimal. Detection threshold \(\tau\) then classifies real vs. hallucinated objects.

Key Contributions

  • First OH detection method to explicitly integrate both global (scene-level) and local (patch-level) embedding similarity, demonstrating their complementary roles.
  • Visual Logit Lens adaptation to multimodal LVLMs for unsupervised object grounding, outperforming attention-based and cosine-similarity-based patch selection.
  • Comprehensive benchmark of OH detection methods across MSCOCO and Objects365 on four LVLMs (LLaVA-1.5-7B/13B, MiniGPT-4, Shikra), with AUROC and AUPR as threshold-independent metrics.
  • Ablation evidence that the last instruction token is the superior global anchor (+8% over average image token), Top-K aggregation beats Top-1, and later-intermediate decoder layers are optimal for embedding extraction.

Results

All values are AUROC (%) unless noted; comparisons are on MSCOCO unless stated.

  • LLaVA-1.5-7B / MSCOCO: GLSim 83.7 vs. SVAR 74.7 (+9.0), Contextual Lens 75.4 (+8.3), Internal Confidence 72.9 (+10.8).
  • LLaVA-1.5-13B / MSCOCO: GLSim 84.8 vs. Contextual Lens 78.7 (+6.1), SVAR 75.2 (+9.6).
  • Shikra / MSCOCO: GLSim 83.0 vs. SVAR 70.7 (+12.3), best prior baseline 70.7.
  • MiniGPT-4 / MSCOCO: GLSim 87.0 vs. Contextual Lens 84.9 (+2.1).
  • Objects365 (harder, 365 classes): GLSim 72.6 AUROC on LLaVA-1.5-7B vs. Internal Confidence 68.7, SVAR 64.9.
  • Ablation: combining global + local (Top-K) exceeds global-only by +4.4% AUROC on LLaVA; VLL grounding beats attention-based grounding by +12.5% for local score alone.

Limitations

  • Scope is limited to object existence hallucination; attribute and relation hallucinations are not addressed due to lack of suitable benchmarks.
  • The VLL-based grounding requires access to intermediate decoder layer activations and the unembedding matrix, making the method intrinsic to transformer LVLMs with accessible internals.
  • Weighting parameter \(w\) and patch count \(K\) require per-model tuning (optimal \(K\) varies: 32 for LLaVA-1.5, 16 for Shikra); layer indices \(l\), \(l'\) also need validation per architecture.
  • Evaluation is limited to open-description prompts; performance under constrained QA or VQA-style prompts is not reported.

Relevance to Vision-Language Models

GLSim directly targets a core reliability problem in LVLMs—object hallucination—using only signals already present in the frozen model's latent space, making it immediately applicable as a post-hoc detector without retraining. The Visual Logit Lens extension to multimodal settings is a general mechanistic interpretability tool that could inform future work on VLM attention and representation analysis. The finding that the final instruction token encodes a compact multimodal scene summary is a concrete insight about VLM internal representations with implications beyond hallucination detection. By providing a rigorous comparative benchmark, the paper also establishes a clearer evaluation framework for the growing OH detection literature.