Skip to content

Groc-PO: Grounded Context Preference Optimization for Truthful Multimodal LLMs

🕒 Published (v1): 2026-07-15 11:28 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Groc-PO addresses the credit-assignment gap in preference optimization for multimodal LLMs by introducing stage-wise supervision across three grounding stages rather than optimizing only at the final-answer level. It constructs a structured 3-stage preference dataset (GCPD) and a dual-weighted loss combining stage-aware and hardness-aware terms. On LLaVA-1.5-7B/13B and Qwen2.5-VL-7B, it consistently outperforms DPO and strong baselines on hallucination and complex-reasoning benchmarks.

Problem

Standard DPO supervises only at the final-answer level, ignoring that multimodal reasoning errors often originate in early visual grounding stages and propagate forward. This indirect credit assignment leaves the upstream grounding stages without stage-specific preference signals, allowing grounding drift and cross-stage error accumulation to degrade final answers even when the DPO loss converges.

Method

Groc-PO builds a three-stage chain-of-thought preference structure. Stage 1 (Object Grounding): a universal prompt extracts entities and attributes; chosen responses are teacher-model generated and iteratively self-corrected, while rejected responses introduce rule-based factual inaccuracies. Stage 2 (Contextual Grounding): reuses human-validated RLHF-V preference pairs covering relational description and VQA, ensuring model-centric distributional alignment. Stage 3 (Grounded Reasoning): a teacher model generates complex inference questions conditioned on prior chosen context; rejected responses come from the target model itself (on-policy) to match its failure distribution. Each stage's prompt is cumulative: \(\text{S3 prompt} = \{I, Q_1, A_1^+, Q_2, A_2^+, Q_3\}\), forcing the model to condition on all prior chosen context.

The Groc-PO loss replaces the uniform DPO objective with a sample-level adaptive weight \(w_i = \lambda_{r(i)} \cdot \gamma_i\):

\[\mathcal{L}_{\text{Groc-PO}} = -\mathbb{E}_{(x_i, y_w, y_l) \sim \mathcal{D}}\left[w_i \cdot \log \sigma(r_i(\theta))\right]\]
  • Stage-aware weight: \(\lambda_r = 1 + \alpha(r-1)\), monotonically increasing with stage index \(r \in \{1,2,3\}\), up-weighting harder later stages.
  • Hardness-aware focusing weight: \(\gamma_i = (1 - \sigma(r_i(\theta)))^\eta\), suppressing gradient on easy-to-distinguish pairs (analogous to focal loss), where \(r_i(\theta)\) is the implicit reward margin between chosen and rejected.

Approximately 12% of chosen samples (~2k) received human-in-the-loop verification by PhD annotators, with ~2% rewritten.

Key Contributions

  • Groc-PO framework: multi-stage preference optimization that provides direct supervision over pre-final grounding stages, mitigating cross-stage error propagation.
  • GCPD dataset: 5,733 images Ă— 3 stages = 17,199 structured preference pairs organized along a perception→understanding→reasoning axis, built atop RLHF-V with teacher-model drafting, iterative self-correction, model-centric sampling, and human verification.
  • Adaptive Groc-PO loss: dual-weighting via stage-aware (\(\lambda_r\)) and hardness-aware (\(\gamma_i\)) terms enabling dynamic allocation of training emphasis by stage difficulty and sample margin.
  • Empirical validation across LLaVA-1.5-7B, LLaVA-1.5-13B, and Qwen2.5-VL-7B on hallucination, general, and complex-reasoning benchmarks.

Results

  • LLaVA-1.5-7B: MM-Hal score 2.76 vs. DPO 2.14 (+29%), MM-Hal hallucination rate 47.0% vs. DPO 58.3%; AMBER-Discrimination Cog↓ 1.5 vs. next-best mDPO 2.4.
  • LLaVA-1.5-13B: MM-Hal score 2.85 vs. DPO 2.47; MM-Hal rate 45.0% vs. DPO 51.0%; AMBER CHAIR 3.8 vs. DPO 6.1; SEED 83.5 vs. RLHF-V 79.2.
  • Qwen2.5-VL-7B: MM-Hal score 3.83 vs. DPO 3.59; MM-Hal rate 0.32 vs. DPO 0.38.
  • Complex tasks (LLaVA-1.5-7B vs. DPO): LLaVA-Bench complex reasoning +45% (56.9→82.5), SEED visual reasoning +35% (38.3→51.8), MM-MT multi-turn +36% (1.88→2.55), MME commonsense reasoning +3%.
  • Ablation: both stage-aware and hardness-aware components individually improve over base DPO (2.46 and 2.42 vs. 2.24 on MM-Hal score); optimal \(\alpha=0.25\), \(\eta=2\).

Limitations

  • GCPD is derived solely from RLHF-V (5,733 images), limiting domain and scale diversity.
  • The three-stage structure and cumulative-context prompting significantly increase data construction cost and complexity relative to standard DPO pipelines.
  • Human-in-the-loop verification covered only ~12% of samples; Stage 3 chosen responses carry the highest residual risk of uncaught errors.
  • Experiments are confined to LLaVA-family and Qwen2.5-VL architectures; generalization to other MLLM families is not demonstrated.
  • Hyperparameters \(\alpha\) and \(\eta\) require tuning; sensitivity analysis shows performance degrades at \(\alpha=0.5\), indicating a narrow effective range.

Relevance to Vision-Language Models

Groc-PO directly targets a structural weakness in preference alignment for VLMs: the mismatch between where errors originate (early grounding) and where DPO applies supervision (final answer). By decomposing the reasoning chain into grounding stages and applying stage-specific preference signals, it advances VLM alignment methodology beyond flat DPO. The focal-loss-style hardness weighting and on-policy rejected-sample construction are transferable design patterns for any MLLM DPO pipeline. Results on Qwen2.5-VL-7B confirm the approach is not architecture-specific, making it relevant to the broader VLM alignment literature targeting hallucination and faithful multimodal reasoning.