Dynamic Multimodal Activation Steering for Hallucination Mitigation in Large Vision-Language Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; dynamic activation steering identifies truthfulness/visual directions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DMAS (Dynamic Multimodal Activation Steering) is a training-free method that reduces hallucinations in Large Vision-Language Models by dynamically steering two distinct sets of attention heads—one governing truthfulness, one governing visual perception—using semantically retrieved and noise-contrast-derived activation vectors respectively. It outperforms existing state-of-the-art methods on MME, POPE, and CHAIR benchmarks without any fine-tuning.

Problem

Existing hallucination mitigation methods for LVLMs are either training-based (costly, architecture-specific) or decoding-based (degrade generation quality). Prior activation-steering approaches like VTI apply fixed, context-agnostic steering vectors, ignoring that truthfulness directions vary significantly across semantic contexts. No prior method jointly and dynamically addresses both truthfulness and visual perception attention mechanisms.

Method

DMAS intervenes in inference-time hidden states via three steps:

1. Truthfulness Steering Vector Database: Using AMBER and SEED datasets, samples are clustered into 4 semantic groups. For each cluster \(C_i\), paired (truthful, hallucinated) responses are fed through the LVLM and the per-cluster truthfulness steering vector is computed as: $\(D_i = \frac{1}{|C_i|} \sum_{j \in C_i} (A_{\text{pos},j} - A_{\text{neg},j})\)$ PCA reduces noise. Cluster centroid embeddings (via all-mpnet-base-v2 sentence transformer) serve as database keys.

2. Visual Perception Steering Vector: A clean image \(V\) and noise-corrupted image \(V'\) (via forward diffusion) are paired with matching/mismatching object descriptions from YOLOv11 detection. The steering vector is \(D_v = A_v - A_{v'}\), again PCA-denoised.

3. Dynamic Intervention: At inference, cosine similarity between the input query embedding and cluster keys retrieves the best-matching truthfulness vector \(D_f\). Top-\(K\) attention heads by activation magnitude are selected via binary masks \(M_f\), \(M_v\), and the hidden state update is: $\(x^{(l+1)} = x^{(l)} + \text{Concat}_{0 \sim H}\left[\text{Attn}^{(l,h)}(x^{(l)}) + \alpha \cdot M_f^{(l,h)} \cdot D_f^{(l,h)} + \beta \cdot M_v^{(l,h)} \cdot D_v^{(l,h)}\right] W_o^{(l)}\)$

Key Contributions

• Empirical finding that truthfulness and visual perception capabilities engage distinct attention head subsets (layer 30 vs. layer 31 in LLaVAv1.5), motivating separate targeted interventions.
• t-SNE visualization showing truthfulness steering vectors cluster by semantic context, motivating dynamic (not static) retrieval.
• A training-free dual-intervention framework combining semantic-retrieval-based truthfulness steering and noise-contrast-based visual perception steering applied to top-\(K\) heads only.
• Demonstrated generalization to out-of-distribution VQA datasets (ScienceQA, ViQuAE).

Results

• MME: +94.66 total score vs. LLaVAv1.5 baseline; +10.89 over prior SOTA ICT. +46 points vs. QwenVL baseline; +5 over prior SOTA VAF.
• POPE (MSCOCO): +5.43% accuracy and +7.14% F1 on LLaVAv1.5; +3.92% accuracy and +5.95% F1 on QwenVL vs. baselines.
• POPE (GQA): +6.94% accuracy, +6.5% F1 on LLaVAv1.5; +6.93% accuracy, +8.26% F1 on QwenVL.
• CHAIR: \(\text{CHAIR}_S\) reduced from 51.0 to 30.8 (−20.2 points); −5 points vs. SOTA VTI on \(\text{CHAIR}_S\); \(\text{CHAIR}_I\) 11.4 (matching VTI at 11.1).
• ScienceQA: 62.27% (LLaVAv1.5) vs. 52.75% baseline; ViQuAE: 56.00% vs. 43.38% baseline.
• Ablation confirms both interventions contribute; truthfulness vector has larger impact than visual perception vector.

Limitations

• Cluster count (4) and hyperparameters (\(\alpha\), \(\beta\), \(K\)) require grid search and may need tuning per model/task.
• Steering vector database is constructed from only two datasets (AMBER, SEED), limiting semantic coverage to their distributions; the 4-cluster granularity is coarse.
• Evaluated on LLaVAv1.5 7B and QwenVL 7B only; scalability to larger or architecturally different LVLMs (e.g., encoder-free models) is untested.
• \(\text{CHAIR}_I\) performance (11.4) does not improve over VTI (11.1), indicating the image-level hallucination gain is marginal.
• The dynamic retrieval step adds inference-time overhead (sentence transformer encoding + cosine similarity over database).

Relevance to Vision-Language Models

DMAS directly targets a core failure mode of LVLMs—object hallucination—without requiring any retraining, making it broadly applicable across model families. The key insight that truthfulness and visual perception engage separable attention head subsets advances mechanistic understanding of multimodal representations in transformer-based VLMs, complementing interpretability work on attention. By demonstrating that context-adaptive (dynamic) activation steering substantially outperforms fixed-vector approaches, this work sets a stronger baseline for inference-time intervention methods and motivates richer steering vector taxonomies. Researchers tracking hallucination mitigation, activation engineering, and LVLM robustness will find this a strong reference point against which future training-free methods should be compared.