Language Bias in LVLMs: From In-Depth Analysis to Simple and Effective Mitigation¶

🕒 Published (v1): 2026-05-24 12:23 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; in-depth analysis + simple mitigation of language bias in LVLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper identifies the root cause of hallucination in Large Vision-Language Models (LVLMs) as modality misalignment during training: both Visual Instruction Tuning (VIT) and Direct Preference Optimization (DPO) inadvertently prioritize textual over visual improvements. The authors formalize this as language bias $B = \log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$ and propose two lightweight loss-level interventions—LBR and LBP—that reduce this bias without extra data or auxiliary models.

Problem¶

Prior work attributes LVLM hallucinations to language bias qualitatively but lacks a formal, quantitative definition or mechanistic understanding of how standard training paradigms cause it. Empirical analyses show that improvements to the multimodal likelihood $\pi_\theta(y|x,v)$ during VIT are largely explained by improvements to the text-only likelihood $\pi_\theta(y|x)$, meaning visual information is underutilized despite being provided.

Method¶

The authors decompose training dynamics by tracking two quantities: the multimodal reward $R = \log\frac{\pi_\theta(y|x,v)}{\pi_{ref}(y|x,v)}$ and the text-only bias $B = \log\frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)}$. They show $R \approx B$ throughout VIT training—quantitative evidence that gains are text-driven. A parallel decomposition applied to DPO shows the text-only gain for preferred responses ($B_{DPO_w}$) outpaces the multimodal gain ($R_{DPO_w}$).

Language Bias Regularization (LBR) is added to the VIT objective: $$\mathcal{L}'_{VIT} = \mathcal{L}_{VIT} + \alpha \cdot B$$ with $\alpha = 10^{-5}$. This penalizes the model for drifting its text-only distribution away from the pre-VIT reference, suppressing linguistic drift.

Language Bias Penalty (LBP) is added to the DPO objective (which already applies to a model carrying accumulated bias): $$\mathcal{L}_{LBP} = -\log\sigma(B), \qquad \mathcal{L}'_{DPO} = \mathcal{L}_{DPOM} + \gamma \cdot \mathcal{L}_{LBP}$$ The sigmoid bounds the penalty to prevent instability, and $\gamma = 1$ suffices across experiments. Both methods plug directly into existing training pipelines.

Key Contributions¶

Formal quantitative definition of language bias as $B = \log(\pi_\theta(y|x)/\pi_{ref}(y|x))$, with empirical evidence it grows monotonically during VIT and DPO.
LBR: a sequence-level L1 regularizer on $B$ for the VIT stage, outperforming L1-mean, KL-divergence, and contrastive alternatives in ablation.
LBP: a sigmoid-shaped penalty on $B$ for the DPO stage, specifically designed for models with pre-existing bias.
No additional data, auxiliary models, or architectural changes required; both methods are drop-in loss modifications.

Results¶

LBR (vs. vanilla VIT baseline on LLaVA-1.5-7B): MME 1490→1525, SQA 66.8→69.4, VisWiz 50.1→54.0, MMMU 35.7→37.2; consistent gains across 10+ benchmarks spanning VQA, captioning, and general understanding.
LBP (vs. V-DPO and MFPO on RLHF-V, 5.7K pairs, LLaVA-v1.5-7B): MMHalBench score 2.91 vs. 2.69 (MFPO), hallucination rate 0.43 vs. 0.49; cuts MMHalBench hallucination rate of baseline LLaVA-v1.5-7B by 27%.
LBP-aligned 7B model matches or exceeds GPT-4V on two sub-tasks of AMBER and Object HalBench.
LBP preserves or improves general benchmark scores (Table 7), avoiding the typical alignment tax.
LBP is robust to $\gamma$ across a wide range; LBR is more sensitive to $\alpha$ but $\alpha = 10^{-5}$ generalizes across all tested models without re-tuning.

Limitations¶

LBR shows only modest gains on dedicated hallucination benchmarks (Object HalBench, MMHalBench), which the authors attribute to CHAIR metric susceptibility to reward hacking rather than method failure.
LBR's hyperparameter $\alpha$ requires careful tuning; very small values ($\alpha = 10^{-7}$) are ineffective.
Evaluation confined to LLaVA family (7B, 13B, 3B); generalization to other architectures (e.g., InternVL, Qwen-VL) is not shown.
Automated hallucination benchmarks are argued to be inadequate for measuring language bias in long-form generation; the human evaluation protocol is described but truncated in the provided text.
Language bias formulation assumes access to a reference model ($\pi_{ref}$), which is always available in the training pipeline but adds a forward-pass overhead.

Relevance to Vision-Language Models¶

This paper provides a principled, quantitative account of why VIT and DPO systematically produce language-biased LVLMs—a question the VLM field has treated empirically until now. The bias metric $B$ is model-agnostic and can serve as a diagnostic for any LVLM training run. The LBR/LBP interventions are minimally invasive and directly applicable to any LVLM trained with standard pipelines, making them practically relevant for the broader community working on multimodal alignment, RLHF, and hallucination reduction.