DAMO: Decoding by Accumulating Activations Momentum for Mitigating Hallucinations in Vision-Language Models¶

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DAMO is a training-free decoding strategy for Large Vision-Language Models (LVLMs) that mitigates hallucinations by injecting a momentum term—analogous to gradient-descent momentum—into the hidden-state update process during the forward pass. The key insight is that LVLMs correctly extract visual information in early layers but "overthink" in later layers, causing abrupt prediction shifts toward visually-ungrounded outputs. DAMO accumulates layer-wise activation deltas into a momentum buffer and uses it to smooth hidden-state updates in those late-layer hallucination surges.

Problem¶

Existing hallucination mitigation methods for LVLMs either (1) require fine-tuning (resource-intensive, risks degrading other capabilities) or (2) apply contrastive decoding only at the final logit layer (VCD, DoLA, OPERA), ignoring the intermediate hidden-state dynamics that cause hallucinations. No prior work corrects hidden states layer-by-layer during the forward pass. The root cause identified here is a localized surge in the last few transformer layers that suppresses visual semantics accumulated in earlier layers—but this surge is not addressed by logit-level interventions.

Method¶

DAMO operates in two coupled mechanisms applied during each autoregressive forward pass, with no additional training:

Activation Momentum (AM): At each transformer layer \(j\), instead of directly adding the raw activation \(f_j(h_j)\) to the hidden state, DAMO maintains a momentum buffer \(v^{j+1} = \beta v^j + (1-\beta)\nabla h^{j+1}\) (where \(\nabla h^{j+1} = f_j(h_j)\) is the raw activation). The hidden state update becomes \(h^{j+1} = h^j + v^{j+1}\), smoothing layer-to-layer updates and preserving early-layer visual context.
Adaptive Activation Refinement: Rather than applying AM from layer 0 (which hurts early visual encoding), DAMO monitors the early-exit probability distribution \(P^j = \text{softmax}(\phi(h^j))\) and tracks its momentum \(\nabla\hat{P}^j\). When the cosine similarity between the current distribution shift \(\nabla P^j\) and the momentum \(\nabla\hat{P}^j\) falls below threshold \(\tau\), a hallucination surge is detected and AM is activated for all remaining layers.
Noise-resistant coefficient switching: Once the surge is detected, the trade-off coefficient switches from \(\beta_1\) (lower, favoring current activations) to \(\beta_2 > \beta_1\) (higher, preserving historical momentum) to prevent hallucinatory late-layer activations from corrupting the momentum buffer.

Key Contributions¶

Empirical characterization of hallucinations as localized late-layer surges: 75% of hallucinated samples show the correct answer peaking mid-network before being suppressed by the final layers.
DAMO: a momentum-based decoding method that corrects hidden states during the forward pass—distinct from all prior logit-contrastive approaches.
Adaptive activation refinement: a data-driven, per-sample trigger for when to begin momentum correction, outperforming any fixed starting layer.
Adaptive coefficient adjustment (\(\beta_1 \to \beta_2\) switching) that makes the momentum buffer resistant to noise injected by hallucinatory layers.
Demonstrated transferability to LLMs (LLaMA2-7B) with gains on TruthfulQA, FACTOR, and GSM8K.
Near-identical memory footprint to regular greedy decoding (e.g., 14.6 GB vs. 22.5 GB for OPERA on LLaVA1.5).

Results¶

MME (perception, 10 tasks):
LLaVA1.5: DAMO 1515.89 vs. Regular 1491.71, OPERA 1518.36, VCD 1458.54
INF-MLLM1: DAMO 1520.74 vs. Regular 1492.71, OPERA 1493.15
mPLUG-Owl2: DAMO 1437.46 vs. Regular 1337.66 (+99.8 pts), VCD 1398.55 (+38.91 pts over VCD)
POPE (MSCOCO, popular setting, mPLUG-Owl2): Accuracy 82.77% vs. Regular 80.73%; F1 83.80% vs. 82.52%
POPE (A-OKVQA, popular setting, mPLUG-Owl2): Accuracy +3.30%, F1 +2.07% over Regular; DOLA gained only +0.16%/+0.09%
LLaVA-Bench (LLaVA1.5): Overall 78.13 vs. Regular 75.22, DOLA 77.17, OPERA 64.17
LLaMA2-7B TruthfulQA MC1: 34.15 vs. Regular 33.66, DOLA 33.29
FACTOR Wiki: DAMO 57.48 vs. DOLA 56.51, Regular 56.91
Adaptive coefficient switching on mPLUG-Owl2 MME: +99.8 total score improvement vs. +48.66 (\(\beta_1\) only) and +90.49 (\(\beta_2\) only)

Limitations¶

Introduces four hyperparameters (\(\tau\), \(\beta_1\), \(\beta_2\), \(\alpha\)) that may require per-model tuning despite reported robustness.
Evaluated only on 7B-parameter LVLMs; scalability to larger models (13B, 70B) is not demonstrated.
The adaptive trigger relies on cosine similarity of distribution shifts, which may be brittle for models with very different layer-distribution dynamics not studied here.
On TruthfulQA MC2 and CoT StrQA/GSM8K, DAMO does not consistently improve over DoLA or Regular, suggesting task-specific limitations in the LLM transfer setting.
No evaluation on open-ended generation quality beyond LLaVA-Bench; long-form captioning or dense VQA not tested.

Relevance to Vision-Language Models¶

DAMO directly targets a fundamental failure mode of current LVLMs—visually-ungrounded generation—from the perspective of internal inference dynamics rather than training data or model architecture. Unlike contrastive decoding approaches (VCD, DoLA) that operate at the final logit level, DAMO intervenes in the residual stream, offering a complementary mechanism applicable alongside other methods. The observation that correct visual information peaks in early-to-mid layers but is suppressed by late-layer processing has implications for understanding how multimodal alignment degrades during deep computation, informing future architectures and training objectives. For researchers tracking VLMs, DAMO provides a computationally cheap, plug-in inference technique applicable to any transformer-based LVLM without retraining.