Cat-PO: Cross-modal Adaptive Token-rewards for Preference Optimization in Truthful Multimodal LLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; cross-modal token-level reward optimization reduces MLLM hallucination

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Cat-PO (Cross-modal Adaptive Token-rewarded Preference Optimization) addresses hallucinations in MLLMs by replacing the uniform token treatment in standard DPO with per-token visual relevance weights derived from three hierarchical signals: global cross-modal attention, local patch entropy, and cross-modal semantic similarity. A token-weighted KL-regularized loss is substituted for the standard DPO objective, yielding fine-grained correction of visually misaligned tokens. Evaluated on LLaVA-v1.5 and Qwen2.5-VL, Cat-PO consistently outperforms prior preference alignment baselines on hallucination benchmarks without requiring any external tools or models.

Problem

Standard DPO-based preference alignment for MLLMs treats all response tokens with identical gradient weight, ignoring that different tokens have widely varying degrees of visual grounding. High-level linguistic tokens (articles, prepositions) carry weak visual signal, while entity and attribute tokens are directly responsible for hallucinated content. Existing token-level alternatives (e.g., TPO, V-DPO) either rely on external reward models, closed-source LLM APIs, or noise injection, rather than exploiting the model's intrinsic multimodal representations. This leads to coarse, resource-intensive corrections that fail to isolate hallucinatory tokens.

Method

Cat-PO operates in four steps, all computed from the MLLM's own internals:

Hierarchical visual relevance scoring for each response token \(y_t\):

1. Global relevance — sum of cross-modal attention scores over all \(N_p\) image patch tokens: $\(S_{\text{global}}(y_t) = \sum_{j=1}^{N_p} a_{t,j}\)$

2. Local relevance — inverse normalized patch-attention entropy (focused attention → high score): $\(S_{\text{local}}(y_t) = 1 - \frac{H(P_t)}{\log N_p}, \quad H(P_t) = -\sum_j P_{t,j}\log(P_{t,j}+\epsilon)\)$

3. Semantic relevance — cosine similarity between the token embedding \(e(y_t)\) and an attention-weighted visual context vector \(v_c(y_t) = \sum_j \alpha_{t,j} v_j\): $\(S_{\text{semantic}}(y_t) = \cos\!\left(e(y_t),\, v_c(y_t)\right)\)$

Unified score fuses the three levels with balance coefficient \(\alpha \in [0,1]\): $\(s_i = \alpha\!\left(0.5\,S_{\text{global},i} + 0.5\,S_{\text{local},i}\right) + (1-\alpha)\,S_{\text{semantic},i}\)$

Token weights apply a smooth \(\tanh\) mapping and base weight \(\lambda_{\text{ref}}\), with opposite polarity for chosen (\(y^+\)) and rejected (\(y^-\)) responses: $\(w_i = \begin{cases} \lambda_{\text{ref}} + \tanh(s_i) & y_i \in y^+ \\ \lambda_{\text{ref}} + 1 - \tanh(s_i) & y_i \in y^- \end{cases}\)$

Cat-PO loss is the sum of a token-weighted DPO objective and a per-token, weight-modulated KL penalty: $\(\mathcal{L}_{\text{Cat-PO}} = \mathcal{L}_{\text{wDPO}} + \lambda\!\left(\sum_t w_t^+ \,\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}} \mid h_t^+) + \sum_t w_t^- \,\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}} \mid h_t^-)\right)\)$

Training uses the RLHF-V dataset (5,733 samples), LLaVA-v1.5-7B/13B and Qwen2.5-VL-3B as base models, 6 epochs, batch size 32, \(\beta_{\text{DPO}}=0.1\), \(\lambda_{\text{KL}}=0.03\).

Key Contributions

• Token-reward mechanism exploiting three levels of intrinsic cross-modal alignment signal, with no external models or tools.
• Smooth \(\tanh\) weighting scheme that asymmetrically boosts visually-aligned chosen tokens and penalizes weakly-grounded rejected tokens.
• Token-level KL regularization that preserves generation quality while suppressing hallucinations at the token level.
• Empirical demonstration that weighting all tokens proportionally to visual relevance strictly outperforms top-\(k\)% subset weighting.

Results

All results on LLaVA-v1.5-7B (RLHF-V data) vs. best prior baselines:

• AMBER-Generation F1: Cat-PO 85.3 vs. TPO 85.0, V-DPO 81.6, DPO 82.1
• MM-Hal Score (↑): Cat-PO 2.74 vs. TPO 2.47, V-DPO 2.16, DPO 2.14
• MM-Hal Hallucination Rate (↓): Cat-PO 42.0% vs. TPO 51.0%, V-DPO 56.0%, DPO 58.3%
• AMBER-Gene CHAIR (↓): Cat-PO 4.8 (ties TPO), vs. DPO 5.7

On LLaVA-v1.5-13B vs. TPO: - MM-Hal Score: Cat-PO 2.85 vs. TPO 2.72; MM-Hal Rate: 42.0% vs. 45.8% - AMBER-Gene CHAIR: 4.3, Hal: 22.0, Cog: 1.6

Qwen2.5-VL-3B generalizes positively over Qwen+DPO on both AMBER-F1 and MM-Hal.

POPE adversarial subset: 84.0% accuracy, 91.3% precision, 82.5% F1 (−2–4% vs. average).

Abstract reports 7–15% improvement on AMBER-Generation and MM-Hal over SOTA.

Ablation confirms both attention-based and similarity-based branches are necessary; the KL term contributes a further gain; simple weighted fusion outperforms a learnable fusion variant (2.74/42% vs. 2.55/50% MM-Hal).

Limitations

• Trained and evaluated exclusively on RLHF-V (5,733 samples); generalization to larger or domain-specific preference datasets is not assessed.
• Learnable multi-branch fusion underperforms the hand-tuned weighted combination, indicating sensitivity to fusion design.
• Hyperparameter sensitivity to \(\alpha\) and \(\lambda_{\text{KL}}\) is reported but automatic tuning is not provided.
• Computational overhead of extracting per-token hierarchical attention scores at every training step is not analyzed.
• Evaluated only on static image VQA; applicability to video MLLMs or multi-image inputs is unaddressed.
• POPE adversarial accuracy degrades 2% vs. average, suggesting residual vulnerability to adversarially constructed hallucination probes.

Relevance to Vision-Language Models

Cat-PO directly targets one of the central reliability challenges in VLMs — hallucination arising from insufficient grounding of generated tokens in visual evidence — and does so by exploiting the cross-modal attention weights already present in standard transformer-based MLLMs (LLaVA, Qwen-VL). This positions it as a lightweight, architecture-agnostic post-training technique applicable to any VLM that exposes cross-modal attention. The method advances the preference optimization line (DPO→TPO→Cat-PO) by introducing the first hierarchical three-level token-reward scheme that jointly uses global attention mass, attention concentration (entropy), and embedding-space semantic alignment, setting a new direction for fine-grained MLLM alignment research. Its no-external-tool constraint makes it practically attractive for production VLM fine-tuning pipelines.