Skip to content

Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ViMaR is a two-stage inference-time decoding framework for VLMs that replaces exhaustive per-step candidate scoring with a coarse caption selection pass followed by targeted segment refinement. A margin-based penalty on low-CLIP-similarity candidates is baked into temporal-difference value training, improving both hallucination rates and caption detail while achieving 4× speedup over the prior VisVM baseline.

Problem

Inference-time search for VLMs (e.g., VisVM) rescores all N candidates at every sentence step, incurring O(N×S) overhead. Simultaneously, standard decoding under-penalizes low-confidence outputs, causing persistent hallucinations. No prior method cleanly addresses both the computational cost and the reward calibration problem together.

Method

ViMaR builds a value model on top of LLaVA-Next-Mistral-7B by attaching a scalar value head to the penultimate transformer layer and training it with temporal-difference (TD) learning. The reward signal uses CLIP-ViT cosine similarity with a margin-adjusted penalty: if CLIP score δ falls below threshold τ, the reward becomes δ − τ (negative penalty proportional to the gap); otherwise it is δ unmodified. The TD loss minimizes (r_si + γ·V(y_{i+1}, I) − V(y_i, I))² over 792K sentence-level triplets derived from 23K COCO images.

At inference, Stage 1 generates N×K full-paragraph candidates via beam search across N temperatures, selects the highest-value caption holistically using Vρ. Stage 2 identifies under-grounded segments in the selected caption (low CLIP alignment or missing objects) and resamples only those segments from the conditional distribution, scoring each alternative with Vρ and inserting the best. This avoids re-scoring the full candidate set on every step.

Key Contributions

  • Margin-based reward adjustment in TD training that explicitly penalizes low-confidence CLIP alignments below a calibrated threshold τ.
  • Two-stage inference: global best-of-N selection + localized segment refinement, reducing total scoring calls vs. VisVM's per-step exhaustive search.
  • Cross-model generalization: a value model trained on LLaVA-Mistral-7B guides decoding on unseen LLaVA-OneVision-Qwen2-7B and Qwen2.5-VL-3B without retraining.
  • Self-training loop: ViMaR-generated captions used as SFT data for LLaVA-Next-7B yield 15.87% average improvement over base model across eight benchmarks.
  • Fully self-contained pipeline using the native CLIP encoder embedded in LLaVA-Next; no external annotations or third-party APIs required.

Results

  • Hallucination (COCO Val2017, n=500): ViMaR achieves CHAIRS=23.1, CHAIRI=4.1, MMHal=3.75, MMHal Rate=0.35 vs. VisVM's 26.2 / 4.6 / 3.30 / 0.39 and greedy's 32.4 / 5.9 / 2.94 / 0.52.
  • Inference speed: ViMaR ~108s/sample vs. VisVM 462s (4.3× faster), CLIP-PRM 286s (2.6× faster), BoN 668s (6.2× faster); greedy is 62s but worst quality.
  • Human preference (n=300 pairs): ViMaR preferred over VisVM 64.0%, over CLIP-PRM 65.3%, over BoN 66.0%, over greedy 69.7%.
  • GPT-4o pairwise: ViMaR preferred 49.3% vs. VisVM, 68.4% vs. CLIP-PRM, 65.4% vs. BoN, 73.8% vs. greedy.
  • Self-training (8 benchmarks): ViMaR SFT data achieves +15.87% avg. over base, +4.79% over VisVM SFT; hallucination rate reduction of 30.87% vs. base (vs. 20.91% for VisVM SFT).
  • Cross-model: ViMaR guidance on LLaVA-OneVision-Qwen2-7B improves MM-Vet 58.8→60.5, MMBench 81.7→84.8; on Qwen2.5-VL-3B, +6.7% average across benchmarks.

Limitations

  • Value model requires LLaVA-Next-Mistral-7B as backbone; porting to a fundamentally different architecture (e.g., encoder-decoder models) may not be straightforward.
  • Stage 2 refinement relies on CLIP similarity to detect under-grounded segments, inheriting known CLIP blind spots (e.g., counting, fine-grained spatial relations).
  • Inference speed (108s/sample) is still far slower than greedy decoding (62s), limiting real-time applicability despite the relative speedup over search baselines.
  • The margin threshold τ is a calibrated hyperparameter; sensitivity analysis and automated selection are not reported.
  • Self-training experiments are confined to LLaVA-Next-Mistral-7B; generality of the self-improvement loop to other base models is not directly evaluated.

Relevance to Vision-Language Models

ViMaR advances inference-time compute scaling for VLMs, a growing paradigm that avoids costly retraining by improving output quality at decoding time. The margin-based TD reward directly addresses the hallucination problem — a central open challenge in VLMs — by shaping the reward landscape during value-model training rather than post-hoc filtering. The cross-model generalization result (value model trained on one architecture guiding a stronger, unseen model) is particularly significant for the field, suggesting inference-time value models can be reused as plug-and-play modules across the VLM ecosystem. The self-training loop also connects to the emerging area of self-improving VLMs, where inference-time search quality determines the ceiling of synthetic data quality.