Hallucination-aware Intermediate Representation Edit in Large Vision-Language Models¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; intermediate representation editing for hallucination mitigation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

HIRE (Hallucination-aware Intermediate Representation Edit) eliminates hallucinations in Large Vision-Language Models (LVLMs) by detecting and editing hallucination-prone intermediate representations at inference time, without retraining model weights or requiring dual forward passes. A dual-encoder Editor learns to separate semantic from hallucinatory components, while a lightweight Router selectively edits only high-risk tokens. A scalar hyperparameter $\alpha$ additionally enables continuous control over hallucination levels.

Problem¶

Existing hallucination mitigation for LVLMs splits into two families, each with critical drawbacks: retraining methods (e.g., HA-DPO, HDPO) demand expensive dataset construction and full/partial fine-tuning; Contrastive Decoding (CD) methods (e.g., VCD, SID, ICD) require two forward passes per inference step—doubling latency—and uniformly adjust all token probabilities regardless of their actual hallucination risk, hurting fluency and wasting compute. Neither family supports controllable hallucination levels, which is desirable for tasks like creative writing.

Method¶

Editor — a dual-encoder autoencoder $G_\phi$ operates on the attention-layer hidden states $h^t_l$ of each transformer layer $l$. A semantic encoder $E_{sem}$ and a hallucinatory encoder $E_{hal}$ decompose the representation into $h^t_{l,sem}$ and $h^t_{l,hal}$. A hallucination-reduction direction is computed as: $$\delta_l = \frac{1}{T}\sum_t (h^{+t}_{l,hal} - h^{-t}_{l,hal})$$ where $+/-$ denote authentic/hallucinated samples. The token-specific editing offset is: $$\Delta^t_l = D\!\left(h^t_{l,sem} + f_{attn}(h^t_{l,sem},\, h^t_{l,hal}+\delta_l)\right) - D\!\left(h^t_{l,sem} + f_{attn}(h^t_{l,sem},\, h^t_{l,hal}-\delta_l)\right)$$

$E_{sem}$ is trained with InfoNCE to preserve token semantics across authentic/noisy pairs; $E_{hal}$ is trained with InfoNCE to separate hallucinated from authentic activations irrespective of token identity. MSE reconstruction and editing losses enforce faithful decoding.

Router — a lightweight MLP $R_\theta$ takes only the first-layer representation $h^t_0$ and outputs a binary decision $c \in \{0,1\}$, gating whether all subsequent layers undergo editing. It is trained from scratch with a reference-free variant of DPO: $$L_r = -\mathbb{E}\!\left[\log\sigma\!\left(\beta\log\pi_\theta(h^+,c^+) - \beta\log\pi_\theta(h^-,c^-)\right)\right], \quad \beta=0.1$$ Preference pairs are ranked by CHAIRI on $N$ candidate captions generated with different editing trajectories.

Inference — the edited representation is: $$h^{t,aug}_l = \begin{cases} h^t_l + \alpha \cdot \Delta^t_l & c=1 \\ h^t_l & c=0 \end{cases}$$ $\alpha \in [-1,1]$ is the Hallucination Regulator: positive $\alpha$ suppresses hallucinations; negative $\alpha$ amplifies them for creative tasks.

Data construction uses MSCOCO training images: authentic representations come from intact images; hallucinated representations come from Gaussian-noised versions. Editor trains on 2,000 samples for 5 epochs; Router trains on 8,000 samples for 100 epochs.

Key Contributions¶

Proposes representation editing as a third hallucination-mitigation paradigm—no weight updates, no dual forward pass.
Dual-encoder Editor that disentangles semantic and hallucinatory subspaces via contrastive learning, computing a layer-wise editing direction $\Delta^t_l$ that is both token-specific and semantics-preserving.
Lightweight Router trained with reference-free DPO to selectively edit only hallucination-prone tokens, eliminating unnecessary computation.
Hallucination Regulator ($\alpha$ hyperparameter) enabling continuous, task-specific control over hallucination level.
State-of-the-art results on CHAIR, POPE, and AMBER with overhead well below CD baselines.

Results¶

CHAIR / LLaVA-1.5 (512 tokens): SHARES 30.2 / CHAIRI 9.7 vs. baseline 51.3 / 16.8; best prior method VTI achieved 35.8 / 11.1. Reduces sentence- and instance-level hallucination by ~41% and ~42% over baseline.
CHAIR / InstructBLIP (64 tokens): CHAIRS 14.8 / CHAIRI 5.4 vs. baseline 25.4 / 8.2; outperforms M3ID+DPO (13.5/5.7 on CHAIRS/CHAIRI — HIRE improves CHAIRI but slightly worse CHAIRS vs M3ID+DPO in this regime).
POPE / LLaVA-1.5: Overall Acc 87.27 / F1 87.23 vs. prior SOTA Octopus 85.79 / 83.44 (+1.48 Acc / +3.79 F1).
POPE / InstructBLIP: Overall Acc 85.27 / F1 85.42 vs. Octopus 84.79 / 83.43 (+0.48 Acc / +1.99 F1).
AMBER / LLaVA-1.5: Generative CHAIR 4.6 / Hal 20.4 vs. Octopus 4.8 / 23.4; discriminative Acc 79.20 vs. Octopus 76.70; AMBER score improves +7.54 over baseline (absolute).
Computational cost: HIRE requires ~11.73 TFLOPs on AMBER vs. ~20 TFLOPs for CD methods (VCD, M3ID, AVISC), staying close to the ~9.89 TFLOP baseline.

Limitations¶

Not truly training-free: requires MSCOCO training data to build Editor and Router, unlike pure inference-time CD methods.
Evaluated only on LLaVA-1.5 and InstructBLIP; no experiments on more recent architectures (InternVL, LLaVA-NeXT, Qwen-VL, etc.).
Router operates solely on the first transformer layer's representation; richer per-layer routing is acknowledged but left to appendix ablations.
The Hallucination Regulator $\alpha$ requires manual tuning for each downstream use case; no automatic calibration is provided.
AMBER composite score for HIRE is not reported in Table 3 (shown as "---"), making direct comparison with Octopus (88.9) unclear from the main paper.

Relevance to Vision-Language Models¶

HIRE directly targets one of the most pressing failure modes in deployed LVLMs—object and attribute hallucination—without touching model weights or inflating inference cost, making it practically adoptable as a plug-in for existing LLaVA-style architectures. The finding that hallucinated and truthful features are linearly separable in intermediate attention representations provides a theoretically grounded mechanism that complements the growing body of mechanistic interpretability work on LLMs. The Hallucination Regulator introduces a new axis of evaluation: not merely "how much hallucination does the method eliminate?" but "how finely can it be controlled?", which is important for researchers studying the tradeoff between faithfulness and creativity in VLM outputs. For those tracking VLMs, HIRE offers an efficient post-hoc approach that can be layered on top of any LVLM without the data cost of DPO/RLHF-based fine-tuning.