Skip to content

Prefill-Time Intervention for Mitigating Hallucination in Large Vision-Language Models

🕒 Published (v1): 2026-04-28 13:42 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Novel prefill-time intervention eliminates LVLM hallucinations; inference-time method

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Prefill-Time Intervention (PTI) mitigates hallucinations in Large Vision-Language Models (LVLMs) by injecting modality-specific steering vectors into the KV cache once during the prefill stage, before any autoregressive error accumulation begins. Unlike decoding-time intervention (DTI) methods that continuously steer hidden states and inadvertently amplify residual "snowball" hallucinations, PTI corrects the initial representations at their source. It achieves state-of-the-art performance on object hallucination benchmarks across multiple LVLMs and is orthogonal to existing decoding-stage methods.

Problem

Decoding-Time Intervention (DTI) methods (e.g., VISTA) reduce hallucination frequency in LVLMs but exacerbate the severity of residual hallucinations via "snowball" cascades: once an erroneous token is generated, continuous steering cannot halt error propagation and each subsequent step compounds the mistake. DTI is also modality-agnostic (uses a single textual steering vector), targets coarse-grained hidden states rather than fine-grained representations, and intervenes reactively after the initial poorly-grounded representations have already been computed.

Method

PTI addresses three failure axes of DTI — when, how, and what to intervene:

  1. When: PTI intervenes exactly once during the prefill stage, directly modifying the initial KV cache \((\mathbf{K}^l, \mathbf{V}^l)\) across all \(L\) decoder layers before decoding starts. No intervention occurs during generation.

  2. What: PTI targets the fine-grained KV cache rather than coarse hidden states, exploiting the decoupled semantic roles of keys (where to attend) and values (what to aggregate).

  3. How (direction extraction): Separate visual and textual steering vectors are extracted from MSCOCO (100 VQA pairs):

  4. Visual: Positive images \(I^i_\text{pos} = I^i \odot M^i_\text{obj}\) (object-only) vs. negative \(I^i_\text{neg} = I^i \odot (1 - M^i_\text{obj})\) (background-only). The steering vectors are averaged differences of KV caches at visual token positions \(\mathcal{I}_\text{img}\), followed by PCA denoising: \(S^l_{k,\text{img}} = \frac{1}{N}\sum_i \text{AP}(K^{i,l}_\text{pos} - K^{i,l}_\text{neg})[\mathcal{I}_\text{img}]\).

  5. Textual: Positive captions retain only object-anchor words; negative captions retain only non-anchor context. Steering vectors are extracted at the last textual token index \(N_x - 1\).

  6. Intervention: At inference, the pre-computed directions are added to the corresponding KV positions: $\(\tilde{K}^l[\mathcal{I}_\text{img}] \mathrel{+}= \lambda_{k,\text{img}} S^l_{k,\text{img}}, \quad \tilde{V}^l[\mathcal{I}_\text{img}] \mathrel{+}= \lambda_{v,\text{img}} S^l_{v,\text{img}}\)$ and analogously for textual positions, with scalar coefficients \(\lambda\) tuned via grid search. The enhanced cache is then the sole initial state for decoding.

Key Contributions

  • PTI paradigm: First prefill-stage, training-free KV cache intervention for hallucination mitigation in LVLMs.
  • Modality-aware, decoupled steering: Derives independent visual and textual directions for keys and values, directly countering modality misalignment.
  • Object-vs-background contrast: Keys are steered toward visually grounded objects; values are steered to filter background noise.
  • Single-shot intervention with negligible overhead: Operates once pre-decoding; introduces no latency during generation.
  • Orthogonal composability: PTI is plug-and-play compatible with existing DTI methods, yielding additive gains.

Results

  • CHAIR (LLaVA-1.5, greedy): PTI reduces \(\text{CHAIR}_S\) from 47.4 (vanilla) to 15.4 (−32.0) and \(\text{CHAIR}_I\) from 13.7 to 5.4 (−8.3), outperforming VISTA (\(\text{CHAIR}_S = 20.4\)).
  • CHAIR (LLaVA-1.5, beam search): \(\text{CHAIR}_S\) reduced from 47.8 to 13.2 (−34.6), best across all baselines.
  • CHAIR (Qwen-VL-Chat, greedy): \(\text{CHAIR}_S\) 39.6 → 20.6 (−19.0); VISTA only reaches 39.0.
  • POPE (LLaVA-1.5, nucleus sampling): PTI achieves average accuracy 82.21% vs. 79.88% vanilla and 81.73% PAI; adversarial accuracy 77.40% vs. 75.40% vanilla.
  • POPE (Qwen-VL-Chat): Average F1 82.85%, surpassing VTI (82.19%) and VISTA (82.12%) which both degrade below vanilla on this model.
  • AMBER (LLaVA-1.5, greedy): \(\text{CHAIR}_I\) drops from 6.1 to 3.8; accuracy improves from 81.5% to 82.0%.
  • PTI demonstrates consistent superiority over PAI, VCD, VTI, OPERA, and VISTA across all three tested architectures (LLaVA-1.5, Qwen-VL-Chat, DeepSeek-VL-Chat).

Limitations

  • Steering direction extraction requires MSCOCO image-caption pairs with object segmentation masks; adapting to domains without such annotations is not discussed.
  • Four scalar hyperparameters (\(\lambda_{k,\text{img}}, \lambda_{v,\text{img}}, \lambda_{k,\text{txt}}, \lambda_{v,\text{txt}}\), reduced to two in practice) require grid search per model, adding a calibration step.
  • Evaluated on three LVLMs; generalization to architectures with fundamentally different visual connector designs (e.g., those without a standard MHSA decoder) is not demonstrated.
  • The paper does not analyze behavior under severe distribution shift between the MSCOCO calibration set and target deployment domains.
  • MMHal-Bench results are shown in a figure (Figure 4) but category-level numbers are not fully reproduced in the provided text excerpt.

Relevance to Vision-Language Models

PTI directly addresses a core reliability failure mode of modern LVLMs — object hallucination and its cascading propagation — by revealing that the KV cache's initial state is the decisive bottleneck, not the decoding procedure. For researchers in VLM alignment and trustworthy perception, the finding that modality-agnostic DTI worsens hallucination severity even while reducing frequency (quantified via PSH) reframes how intervention timing and granularity should be designed. The modality-decoupled, KV-level steering framework is architecture-agnostic and composable, making it directly applicable to current instruction-tuned VLMs (LLaVA, Qwen-VL, DeepSeek-VL). This work sits at the intersection of KV cache manipulation — an active area for reasoning and long-context research — and visual grounding, pointing toward prefill-stage representation shaping as a promising design axis for future VLMs.