Skip to content

Mitigating Hallucination in Vision-Language Model with Depth and Spatial-aware Key-Value Refinement

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; depth and spatial-aware KV cache refinement reduces VLM hallucination

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Visual hallucinations in VLMs arise when key vectors scatter isotropically in the KV cache, severing cross-modal attention from image content. DSCR (Depth and Spatial-aware Cache Refinement) is a training-free, plug-and-play method that injects monocular depth and 2D spatial proximity into the visual KV cache at inference time to restore coherent key-vector clustering. It yields up to 41.6% accuracy gains across multiple hallucination benchmarks without modifying model weights.

Problem

Existing hallucination mitigations (extra visual supervision, RL reward modeling, attention reshaping) lack a mechanistic explanation for why VLMs hallucinate. The representational root cause—incoherence among key vectors of adjacent visual tokens—had not been identified or directly corrected, leaving a gap for a principled, geometry-aware inference-time fix.

Method

DSCR operates exclusively on the frozen visual KV cache immediately before text decoding.

  1. Depth estimation: An off-the-shelf monocular depth estimator produces a depth map \(D \in \mathbb{R}^{H \times W}\), min-max normalized to \([0,1]\); depth values \(\{d_i\}\) are aligned to patch resolution.

  2. Proximity weights: For every patch pair \((i,j)\), a combined proximity score is computed: $\(\tilde{w}_{ij} = f_d(d_i - d_j)^\alpha + f_s(\mathbf{s}_i - \mathbf{s}_j)^\beta\)$ where \(f_d(\cdot) = \exp\!\bigl(-\tfrac{(d_i-d_j)^2}{2\sigma_d^2}\bigr)\) (depth-to-depth Gaussian) and \(f_s(\cdot) = \exp\!\bigl(-\tfrac{\|\mathbf{s}_i-\mathbf{s}_j\|_2^2}{2\sigma_s^2}\bigr)\) (2D spatial Gaussian). Fixed hyperparameters: \(\sigma_d=\sigma_s=0.6\), \(\alpha=0.6\), \(\beta=0.8\). The diagonal is masked (\(\tilde{w}_{jj}=0\)) and weights are \(\ell_1\)-normalized.

  3. Cache update: Every visual key and value entry is replaced by a weighted average: $\(\hat{\mathbf{k}}_j^I = \sum_i w_{ij}\mathbf{k}_i^I, \quad \hat{\mathbf{v}}_j^I = \sum_i w_{ij}\mathbf{v}_i^I\)$ implemented as a single matrix multiplication. The theoretical grounding interprets this as graph-Laplacian smoothing on the token graph: tokens on the same surface are clustered into a low-frequency subspace; tokens across depth discontinuities are pushed apart.

Key Contributions

  • KV-coherence hallucination theory: PCA visualization and attention-score diagnostics reveal that isotropic key-vector scatter at "suspect tokens" causes hallucination by degrading cross-modal attention; coherent adjacent-token alignment prevents it.
  • DSCR method: First use of KV-cache refinement (as opposed to logit adjustment or attention masking) for VLM hallucination mitigation; training-free, model-agnostic, query-agnostic, adds negligible inference cost.
  • Depth-sensitive benchmark: A 50-image mini-benchmark targeting occlusion and similar-depth scenes with four depth/size reasoning questions per image.
  • Composability: DSCR stacks additively on top of VCD, OPERA, HALC, DAMO, and AGLA, consistently providing extra gains.

Results

  • MME hallucination subset (LLaVA-1.5): 892.68 → 925.96 (+33.28 pts) over baseline; +11.27 over VCD; +53.76 over DAMO; +26.53 over HALC.
  • MME (Qwen-VL): 825.01 → 872.61 (+47.60 pts).
  • Depth hallucination mini-benchmark: +41.4% accuracy over conventional methods.
  • MME position subset: +6% accuracy.
  • POPE (GQA, LLaVA-1.5): F1 gains of +0.03 (Random), +0.03 (Popular), +0.03 (Adversarial).
  • RePOPE (MSCOCO, LLaVA-1.5): F1 gains of +0.08 (Random), +0.05 (Popular), +0.05 (Adversarial).
  • CHAIR (LLaVA-1.5): CHAIR\(_I\) 12.5 → 11.2 (lowest among baselines); CHAIR\(_S\) matches OPERA at 39.2.
  • AMBER: CHAIR\(_S\) reduced ~6.35%; overall F1 improved ~14.2% vs. baseline.
  • Claimed up to 41.6% accuracy gains across all benchmarks combined.

Limitations

  • Relies on an external monocular depth estimator; depth quality degrades on textureless, reflective, or indoor-scene images, potentially introducing estimation errors into the KV cache.
  • Hyperparameters (\(\sigma_d, \sigma_s, \alpha, \beta\)) are fixed and not validated to generalize across all image distributions or model families.
  • Depth mini-benchmark is small (50 images, 4 questions each); generalization claims for depth-sensitive scenarios rest on limited data.
  • Evaluated primarily on LLaVA-1.5/1.6, Qwen-VL, and mPLUG-Owl2; coverage of more recent frontier VLMs (GPT-4o, Gemini, Claude-level architectures) is not reported.
  • The spatial smoothing may over-cluster tokens at object boundaries that coincide with similar-depth regions, potentially blurring fine-grained spatial distinctions.

Relevance to Vision-Language Models

DSCR directly addresses the persistent object hallucination problem in VLMs by targeting a previously underexplored internal mechanism—KV-cache incoherence—rather than output distributions or training data. The training-free, plug-and-play design means it can be adopted immediately on deployed VLMs without retraining, which is highly relevant for practitioners. It establishes KV-vector coherence as a diagnostic lens for visual grounding quality, complementing attention-based interpretability work. For researchers tracking VLMs, this offers both a practical anti-hallucination primitive and a new analytical framework (PCA of key vectors per layer) for studying cross-modal attention failures.