SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineering¶

🕒 Published (v1): 2026-07-05 08:07 UTC · Source: Arxiv · link

Why this paper was selected

Visual token engineering to mitigate hallucinations in LVLMs — practical LVLM reliability fix

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SeeMe is a training-free framework that reduces hallucinations in Large Vision-Language Models (LVLMs) by restructuring—rather than suppressing—visual tokens through a three-stage pipeline of cross-modal pruning, similarity-guided fusion, and attention-based refinement. Borrowing the concept of feature engineering from classical ML, it builds an enriched visual token pool and then selects only the tokens most aligned with the textual query, improving visual grounding without any architectural changes or retraining. Experiments across four 7B-parameter LVLMs on MME, POPE, and AMBER consistently show gains over strong decoding-stage baselines.

Problem¶

Existing hallucination-mitigation methods (DoLa, VCD, DAMO, DCLA) intervene in the language decoder's internal dynamics but overlook a primary upstream cause: irrelevant or noisy visual tokens from the vision encoder that mislead generation. Token suppression methods (EAZY, SPIN) face a hard trade-off—aggressive removal of noisy tokens risks discarding fine-grained visual evidence. No prior work attempts to reconstruct a higher-quality visual token representation before decoding.

Method¶

SeeMe applies three sequential stages inside the frozen decoder of an LVLM, treating visual tokens as editable features (analogous to feature engineering in tabular ML):

Stage 1 – Cross-Modal Attention Selector. At decoder layer $K-1$, cross-modal attention weights from the concatenated query+response $\hat{H}_q = [H_q; H_{\text{resp}}] \in \mathbb{R}^{(L_q+L_o)\times d}$ to visual tokens $H_v \in \mathbb{R}^{L_v \times d}$ are computed: $$C_{K-1} = \text{Softmax}\!\left(\frac{\hat{H}_q H_v^\top}{\sqrt{d}}\right).$$ Each visual token receives an importance score $r_i = \frac{1}{L_q+L_o}\sum_j C_{K-1}[j,i]$, and the top-$k_1 = r \cdot |Z_v|$ tokens are retained as $Z_v^{\text{sel}}$.

Stage 2 – Similarity-Guided Token Expander. Starting from the original embedding-layer visual tokens (not the pruned set, to avoid re-injecting noise-discarded tokens), $\ell_2$-normalized cosine similarity $S = \frac{X}{\|X\|_2}\left(\frac{X}{\|X\|_2}\right)^\top$ identifies the top-$t$ nearest neighbors $\mathcal{N}_i$ for each token $x_i$. A weighted fusion produces new tokens: $$\hat{x}_i^{(j)} = \alpha_{ij}\cdot x_i + (1-\alpha_{ij})\cdot x_j, \quad \alpha_{ij} = \frac{s_{ij}}{s_{ij}+1}.$$ Fused tokens are concatenated with the originals to form an enriched pool $Z_v^{\text{enh}} = [Z_v^{\text{fused}}; Z_v]$.

Stage 3 – Cross-Modal Token Refiner. Self-attention over the full sequence at layer $K$ computes how much each fused token is attended to by textual tokens. The top-$n$ fused tokens by this cross-modal score are selected and appended to $Z_v^{\text{sel}}$, yielding the final sequence $\tilde{Z} = [X_s; Z_v^{\text{final}}; Z_v^{\text{sel}}; X_q]$.

Key Contributions¶

First application of the classical feature engineering paradigm (expand-then-compress) to LVLM visual tokens, proposing reconstruction over suppression.
SeeMe, a training-free, plug-and-play three-stage framework requiring no fine-tuning or architectural changes.
Comprehensive evaluation on MME, POPE, and AMBER across four diverse 7B LVLMs (LLaVA-1.5, LLaVA-NEXT, INF-MLLM, mPLUG-Owl2).
Ablation analysis showing Stage 3 is critical and that similarity-guided fusion outperforms naively re-appending pruned tokens.

Results¶

MME (Perception + Cognition Total): - LLaVA-1.5: SeeMe 1829.49 vs. Regular 1785.85, DCLA 1800.14, SPIN 1787.62 (best) - LLaVA-NEXT: SeeMe 1873.77 vs. Regular 1849.30, DCLA 1855.73, SPIN 1832.44 (best) - INF-MLLM: SeeMe 1780.46 vs. Regular 1758.03, DCLA 1782.26 (second-best; DCLA edges it on total but SeeMe matches on Cognition) - mPLUG-Owl2: SeeMe 1817.75 vs. Regular 1805.25, SPIN 1797.94 (best)

POPE (average Accuracy/F1 across Random/Popular/Adversarial splits): - On MSCOCO, LLaVA-1.5: SeeMe Accuracy 86.07 / F1 86.73 vs. DCLA 85.82/86.45 (best) - On A-OKVQA, INF-MLLM: SeeMe Accuracy 84.70 / F1 86.01 vs. DCLA 84.14/85.91 (best) - SeeMe achieves best or near-best accuracy and F1 across all four models on all three POPE dataset splits.

Baselines compared: Regular decoding, VCD, DoLa, DCLA, SPIN.

Limitations¶

Introduces three additional per-layer operations (attention extraction, cosine similarity matrix, self-attention scoring), adding inference latency not quantified in the paper.
Hyperparameters (pruning ratio $r$, neighbor count $t$, retention count $n$, target layer $K$) are model-specific and require per-model tuning; sensitivity analysis is limited to Stage 3 token count.
Evaluated only on 7B-parameter models; generalization to larger-scale or proprietary LVLMs is untested.
AMBER results are referenced in the text but full numerical tables are not included in the provided excerpt.
The method operates on a single decoder layer; optimal layer selection strategy is not systematically ablated.

Relevance to Vision-Language Models¶

SeeMe directly addresses a structural weakness in the standard LVLM pipeline—the quality of visual tokens passed from the vision encoder to the language decoder—rather than patching decoder-side behavior after the fact. This makes it complementary to, and composable with, existing decoding interventions, which is valuable for practitioners building on LLaVA-style architectures. The expand-then-compress token engineering paradigm it introduces offers a reusable design principle for future work on efficient visual context compression. By demonstrating consistent gains across four architecturally diverse models, it provides evidence that visual token noise is a generalizable source of hallucination rather than an artifact of any single model's design.