Reducing Hallucinations in Large Vision-Language Models via Latent Space Steering¶

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Hallucinations in large vision-language models (LVLMs) often stem from instability in vision encoder features—approximately 15% of features exhibit high variance under mild image perturbations, which the text decoder amplifies into hallucinated outputs. This paper proposes Visual and Textual Intervention (VTI), a training-free, test-time method that steers latent representations in both the vision encoder and text decoder toward more stable, less-hallucinated directions using PCA-derived direction vectors precomputed from 80 COCO examples. VTI outperforms prior decoding-based methods across POPE, CHAIR, and MMHAL-Bench on multiple LVLMs.

Problem¶

Existing LVLM hallucination mitigations (VCD, OPERA, DoLA) target only the text decoding process and do not address the root source of cross-modal misalignment: the sensitivity of the text decoder to unstable vision encoder outputs that arise when the two components are pre-trained separately and only lightly fine-tuned together. Prior mechanistic analyses (language priors, statistical bias) also do not account for the LVLM-specific encoder→decoder sequential structure.

Method¶

VTI operates in two steps:

Direction precomputation (offline, once): For each of N=80 reference images, apply m random masks to produce perturbed copies, run the vision encoder on each, and compute the per-layer, per-token average hidden state h̄. The visual shifting vector Δv = h̄ − h is computed for each example; PCA across all N examples extracts the first principal component d_vision_{l,t}, removing image-specific noise and retaining the dominant "stabilization direction." Separately, hallucinated captions are generated by GPT from clean captions; the textual shifting vector d_text_{l,t} is the PCA direction of (hidden state of clean caption) − (hidden state of hallucinated caption) at the final token of the text decoder.
Test-time intervention: For every new query, add α·d_vision to all vision encoder hidden states at every layer and token position, and β·d_text to all text decoder hidden states at every layer (last token only, since the decoder is causal). α, β ∈ [0.1, 1.0] are selected by grid search. No additional forward passes beyond one are needed at inference.

Key Contributions¶

Empirical demonstration that ~15% of vision encoder features are unstable under mild perturbations and that this instability correlates with hallucination rate (averaging across perturbations reduces CHAIRs monotonically).
VTI: a task- and dataset-agnostic test-time latent space steering method requiring no retraining, no auxiliary models, and only 80 calibration examples fixed across all benchmarks.
Decomposition showing visual and textual interventions are complementary: visual shifting improves image-level (CHAIR_I) and attribute/comparison tasks; textual shifting improves sentence-level (CHAIR_S) and language-reasoning tasks (counting, adversarial).
Evidence that VTI improves feature stability (lower variance across perturbation types) without the information loss incurred by naive feature averaging (linear probe accuracy maintained vs. ~62% for direct averaging).

Results¶

POPE (LLaVA-1.5): VTI 86.5% accuracy / 85.9 F1 vs. OPERA 84.2/83.7, VCD 82.3/83.4, Vanilla 79.8/79.4.
POPE (InstructBLIP): VTI 81.8/83.2 vs. OPERA 79.6/80.9, VCD 80.1/81.0, Vanilla 76.3/78.0.
POPE (Qwen-VL): VTI 85.2/84.1 vs. OPERA 84.3/82.6, VCD 84.5/83.3, Vanilla 83.5/81.2.
CHAIR (LLaVA-1.5): VTI CHAIR_S 35.8 / CHAIR_I 11.1 vs. OPERA 47.0/14.6, VCD 51.0/14.9, Vanilla 51.0/15.2; Recall maintained at 76.8.
CHAIR (InstructBLIP): VTI CHAIR_S 43.4 / CHAIR_I 11.8 vs. OPERA 54.0/12.8, VCD 57.0/17.0, Vanilla 54.0/18.1.
MMHAL-Bench (LLaVA-1.5): VTI average score 2.90 / hallucination rate 0.51 vs. OPERA 2.64/0.59, VCD 2.69/0.58, Vanilla 1.99/0.62. VTI leads on 7 of 8 hallucination categories.

Limitations¶

Hyperparameters α and β require grid search per model; optimal values may vary across architectures and tasks.
Direction precomputation relies on paired hallucinated/non-hallucinated captions generated by GPT, introducing a dependency on an external LLM.
Evaluated only on three LVLMs (LLaVA-1.5, InstructBLIP, Qwen-VL); generalization to newer architectures (e.g., those with tighter vision-language co-training) is untested.
VTI adds fixed direction vectors uniformly across all layers/tokens; layer-wise or token-selective application could potentially improve or degrade performance in ways not explored.
Calibration set of 80 COCO images may not cover domain-shifted or specialized visual distributions.

Relevance to Vision-Language Models¶

VTI directly targets a structural weakness unique to LVLMs—the separately pre-trained encoder/decoder pipeline—distinguishing it from LLM hallucination mitigations and providing mechanistic insight into why cross-modal misalignment occurs. As a training-free, test-time method with a fixed 80-example calibration overhead, it is immediately applicable to any LVLM without access to training data or model weights beyond inference. The representation engineering perspective (latent space steering via PCA directions) bridges LLM interpretability work (representation engineering) with multimodal reliability, suggesting a productive direction for fine-grained control of LVLM outputs. This is directly relevant for practitioners deploying LVLMs in reliability-sensitive applications and for researchers studying the role of vision feature quality in multimodal generation fidelity.