Skip to content

Image Token Matters: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LVLMs with discrete image tokenizers hallucinate partly because co-occurring image tokens in the codebook create "visual priors" that evoke absent tokens during generation. This paper introduces CGC+VTD: a GNN-based clustering method to identify hallucination-prone absent tokens, followed by a latent-space subtraction to suppress their influence at inference time. The approach outperforms prior hallucination mitigation baselines across three models and three benchmarks with the lowest computational overhead.

Problem

Existing hallucination mitigation methods (contrastive decoding, OPERA, PROJECTAWAY) were designed for continuous-feature LVLMs like LLaVA and underperform or degrade when applied to discrete tokenizer-based LVLMs (Chameleon, Emu3, Janus-Pro). The root cause specific to discrete tokenizers—visual priors encoded in codebook token co-occurrence patterns—is unaddressed.

Method

Context-Guided Clustering (CGC): A co-occurrence graph is built over codebook tokens using COCO 2017 Panoptic images. Edge weights combine spatial proximity (3×3 grid neighborhood) and semantic coherence (same segmentation mask). A GNN with attentional aggregation is trained using a contrastive InfoNCE loss (positive pairs weighted by co-occurrence strength) plus a positive-pair similarity loss (Lpps) that enforces absolute similarity thresholds. Top-10% token pairs by co-occurrence strength form edges. K-means on the learned GNN embeddings produces clusters reflecting visual priors. CGC runs once offline per model.

Visual Token Decontamination (VTD): At inference time, the dominant clusters (those most frequently represented in the input image's tokens) are identified. Tokens in those clusters that are absent from the input image are treated as hallucination-inducing. For each absent token vabs, VTD subtracts its projection from the hidden representation of each present image token at transformer layer l:

g(l)(vi) := g(l)(vi) − γ · [g(l)(vi)·g(l)(vabs) / ||g(l)(vabs)||²] · g(l)(vabs)

The editing layer l, coefficient γ, and number of dominant clusters are hyperparameters tuned per model on an AMBER subset.

Key Contributions

  • First identification and quantification of visual priors in discrete image tokenizer codebooks as a source of object hallucination in LVLMs.
  • CGC: a GNN + contrastive learning + K-means pipeline that outperforms naive K-means in capturing co-occurrence-based visual priors (higher HitRate@K for absent tokens in dominant clusters).
  • VTD: a training-free, single-pass latent editing intervention that suppresses hallucinative absent tokens during autoregressive decoding.
  • Empirical evidence that absent tokens in dominant clusters have 5–10% higher HitRate association with hallucinated objects than present tokens or tokens from non-dominant clusters.
  • Demonstrated complementarity with OPERA (orthogonal modality targets), SID, and PROJECTAWAY.

Results

  • AMBER + Object HalBench (Chameleon-7B): CGC+VTD achieves best/second-best in 22/25 metrics. CHAIR drops 5.46 points vs. Nucleus Sampling; Cog improves by 10.82 points; Object HalBench CHAIR-s: 38.58 vs. 46.15 (OPERA) baseline.
  • Object HalBench (Janus-Pro-7B): CHAIR-s drops 8.9 points (19.33→10.43) and CHAIR-i drops 4.87 points (10.68→5.81) vs. Nucleus Sampling; best result among all baselines.
  • MME: CGC+VTD maintains or improves general perception—outperforms all baselines on Janus-Pro-7B (1813.93) and Emu3-13B (1635.71).
  • Efficiency (Chameleon-7B): 188s vs. 941s (OPERA, 5× faster) and 470s (PROJECTAWAY, 2.5× faster); GPU memory 14,478 MB (lowest, tied with PROJECTAWAY).
  • Combination: CGC+VTD + OPERA improves CHAIR-s to 33.50 (−6.06 vs. OPERA alone) and CHAIR-i to 22.70 (−2.56) on Object HalBench.
  • Ablation: Removing semantic coherence, spatial proximity, or Lpps each degrades CHAIR-s/i across all three models, confirming all three components are necessary.

Limitations

  • Slight drop in object coverage (Cover↑ score) across models, indicating a hallucination–coverage trade-off.
  • Restricted to LVLMs with discrete image tokenizers; not applicable to continuous-feature models (e.g., LLaVA).
  • Hyperparameters (editing layer, γ, number of dominant clusters) require per-model tuning on a held-out subset.
  • CGC relies on COCO 2017 Panoptic for graph construction; generalizability to out-of-distribution visual domains is not evaluated.

Relevance to Vision-Language Models

This paper targets a hallucination mechanism unique to the emerging class of unified discrete-token LVLMs (Chameleon, Emu3, Janus-Pro), which are increasingly important as the field moves toward single-codebook multimodal architectures. It demonstrates that existing contrastive decoding methods—developed for continuous-feature models—are ineffective or harmful in this new paradigm, motivating tokenizer-aware mitigation strategies. The CGC+VTD framework is training-free, efficient, and composable with language-side methods, making it practically deployable. For researchers tracking VLMs, this highlights that discrete tokenization introduces qualitatively new failure modes (codebook co-occurrence priors) that require distinct analysis and intervention.