Skip to content

HIVE: Understanding Post-Hallucination Reasoning in Vision Language Models

🕒 Published (v1): 2026-07-08 15:02 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; dissects post-hallucination reasoning chains in VLMs, not just detection

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HIVE is an evaluation infrastructure that studies Post-Hallucination Reasoning (PHR)—what happens to VLM predictions after hallucinated semantics enter the inference context—rather than how to suppress hallucinations. Across 9 tasks and 9 models, hallucinated captions consistently improve accuracy on vision-language benchmarks while showing limited and unstable effects on text-only tasks. Mechanistic analysis reveals that hallucinated captions broaden semantic coverage, widen embedding distributions, and modulate reasoning entropy in ways correlated with correct predictions.

Problem

Prior work on VLM hallucinations focuses exclusively on detection and mitigation at generation time, leaving the downstream reasoning stage unexplored. It is unknown whether hallucinated semantics, once present in the model's context, help, hinder, or simply perturb subsequent reasoning—a gap that matters for reliability and interpretability of multimodal systems.

Method

HIVE constructs paired faithful (\(C_F\)) and hallucinatory (\(C_H\)) captions from identical generation configurations (same prompt, temperature, token budget), then routes each through a majority-voting ensemble discriminator to retain only high-confidence \(\langle C_F, C_H \rangle\) pairs. Downstream task accuracy is measured under three controlled conditions:

\[y_{\text{raw}} = f(x), \quad y_F = f(x \| C_F), \quad y_H = f(x \| C_H)\]

The hallucination effect is quantified as:

\[\Delta(H{-}F) = \mathbb{E}_{(x,y)\sim\mathcal{D}}\left[\mathcal{L}(f(x\|C_H), y) - \mathcal{L}(f(x\|C_F), y)\right]\]

Mechanistic analysis operates at three levels: embedding distribution shifts (caption-level), step-wise intra-chain cosine similarity (reasoning-process level), and entropy analysis of correct vs. incorrect predictions (output level).

Key Contributions

  • Identifies and formalizes Post-Hallucination Reasoning (PHR) as a distinct, overlooked stage in VLM inference.
  • Introduces HIVE, a controlled evaluation infrastructure supporting caption generation, ensemble-based hallucination discrimination, and task-solver evaluation across matched conditions.
  • Provides mechanistic analysis showing hallucinated captions (i) broaden semantic coverage and widen embedding distributions, (ii) modulate reasoning entropy correlated with correct predictions, and (iii) do not disrupt intra-chain convergence or inter-chain consistency.

Results

  • Vision-language tasks (Table 2): Hallucinated captions yield consistent accuracy gains across all four VL datasets and all four VLMs tested:
  • ISIC (dermatology): GPT-4o +11.76 pp, Qwen-VL-Max +16.91 pp
  • PlantVillage (agriculture): Claude 3 Sonnet +17.22 pp, GPT-4o +14.68 pp
  • Dex-Net (robotics): GPT-4o +2.51 pp, Qwen-VL-Max +2.58 pp
  • GQA: GPT-4o +2.86 pp (mixed results for other models)
  • Text-only tasks (Table 1): Effects are small and inconsistent across 7 LLMs on 5 datasets; many show drops (e.g., DeepSeek v3 on BBBP: −6.04 pp, O3 on BBBP: −13.80 pp).
  • Hallucination discriminator reliability: 81.76% on TruthfulQA; 83.72% on 180 manually annotated captions.
  • Random-checker control (Table 3): Replacing the discriminator with random filtering yields 0.17–1.23% deltas, all non-significant (\(p > 0.14\)), confirming the discriminator is load-bearing.
  • Token-level ablation (Table 5): Masking hallucinated evidence tokens causes large accuracy drops on cases where only the hallucinated path succeeds (e.g., ISIC: H-after 0.380, PlantVillage: H-after 0.700 vs. near-perfect H-before).
  • Reasoning convergence: Inter-chain pairwise similarity ≈ 0.97 for both H and F conditions, with no significant difference (\(p > 0.6\)), showing hallucinations preserve inference stability.

Limitations

  • Study is purely observational/evaluative; HIVE does not propose a training method or mitigation strategy based on PHR findings.
  • Hallucination discriminator accuracy (~82–84%) introduces residual label noise; though controlled via ensemble voting, some mis-labeled pairs may remain.
  • Vision-language experiments cover only four models (GPT-4o, Claude 3 Sonnet, Gemini 2.0 Flash, Qwen-VL-Max) and four datasets; generalizability to open-source or smaller VLMs is not demonstrated.
  • Mechanistic analyses (embedding shifts, entropy) are correlational, not causal; the paper does not identify which internal components drive PHR.
  • The paper text is truncated before the caption embedding distribution figure is described in full, suggesting some analysis details are missing from the provided excerpt.

Relevance to Vision-Language Models

HIVE challenges the dominant framing that VLM hallucinations are purely harmful by demonstrating that hallucinated semantics can act as speculative semantic anchors that expand the hypothesis space and improve downstream task accuracy, especially in perception-heavy settings with partial observability. This has direct implications for how the community evaluates and designs VLM inference pipelines: accuracy on a downstream task is not a reliable proxy for perceptual faithfulness. For researchers studying VLM robustness, reasoning interpretability, or multimodal chain-of-thought, HIVE provides a reproducible controlled-intervention framework to probe how intermediate semantic content—faithful or not—shapes inference trajectories. The modality-dependent pattern (VL tasks benefit; text-only tasks do not) also narrows the conditions under which hallucination-augmented prompting might be exploited or must be guarded against.