Skip to content

Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SpatialReasoner-R1 is a VLM trained with fine-grained Direct Preference Optimization (fDPO) that applies segment-specific learning signals to the descriptive grounding and logical reasoning components of Long Chain-of-Thought (LongCoT) traces. Training data is generated via Multi-Model Monte Carlo Tree Search (M3CTS), which leverages multiple VLMs to produce diverse, reward-verified reasoning trajectories. The result is a new state-of-the-art on SpatialRGPT-Bench, outperforming SpatialRGPT-8B by 9.4% average accuracy while also improving on general VL benchmarks.

Problem

Current VLMs fail at fine-grained spatial reasoning from 2D images—tasks requiring multi-step logic, precise object localization, depth-aware distance estimation, and relative orientation understanding. Standard DPO treats entire LongCoT responses with a single global trade-off parameter β, causing the model to overfit on easier descriptive segments while under-optimizing harder logical reasoning steps. High-quality LongCoT training data for spatial reasoning is also scarce.

Method

fDPO (fine-grained Direct Preference Optimization): Each LongCoT response is segmented into a <description> block (Rdesc) and a <reasoning> block (Rreason). Separate preference differentials ΔRdesc and ΔRreason are computed from composite reward scores for each segment. Softmax-normalized segment weights (controlled by sensitivity λ) are mapped to segment-specific trade-off parameters βdesc and βreason via βs = β(1 + α(2ws − 1)), where α caps the scaling amplitude. The final objective sums the two segment-specific DPO log-ratio terms before applying sigmoid loss.

Fine-grained Spatial Rewards: Four scalar rewards evaluate candidate reasoning paths: (1) Visual Consistency Reward (Rvc)—checks object presence and attribute fidelity against the RGB image; (2) Depth-Guided Spatial Reward (Rsp)—validates spatial assertions against both RGB and monocular depth maps with uncertainty and context-aware weighting; (3) Logical Coherence Reward (Rlc)—verifies multi-hop inference integrity and premise consistency. Composite scores: score(Rdesc) = Rvc + Rsp,desc; score(Rreason) = Rlc + Rsp,reason.

M3CTS (Multi-Model Monte Carlo Tree Search): Multiple VLMs concurrently expand reasoning states; candidates are scored across visual, spatial, and logical dimensions by M evaluation models (+1/0/−1 indicators); low-scoring paths are pruned; value estimates are backpropagated; UCB selects the next expansion node. This produces positive/negative LongCoT pairs for fDPO training.

Key Contributions

  • fDPO: segment-level DPO with dynamically adaptive βdesc and βreason, addressing the under-optimization of logical reasoning segments in standard DPO.
  • Fine-grained spatial reward mechanism: three-dimensional evaluation (visual consistency, depth-guided spatial, logical coherence) to score and select preference pairs.
  • M3CTS: multi-VLM MCTS pipeline for generating diverse, reward-aligned LongCoT training data for spatial reasoning.
  • SpatialReasoner-R1: an 8B/4B VLM (built on a region-prompted architecture with image encoder + text decoder + visual prompt masks) achieving new SoTA on SpatialRGPT-Bench without auxiliary modalities at inference.

Results

  • vs. SpatialRGPT-8B (prior SoTA specialized model): SpatialReasoner-R1 fDPO 8B gains +2.9% qualitative accuracy and +15.8% quantitative accuracy on SpatialRGPT-Bench; overall average +9.4%.
  • fDPO vs. standard DPO (8B): +4.1% on qualitative tasks, +9.0% on quantitative tasks.
  • Quantitative task breakdown (fDPO 8B vs. DPO 8B): Direct Distance 70.95% vs. 62.83%; Horizontal Distance 72.13% vs. 56.55%; Vertical Distance 74.52% vs. 60.37%; Width 80.45% vs. 70.45%; Height 74.43% vs. 68.42%; Direction 94.39% vs. 93.45%.
  • Qualitative task breakdown (fDPO 8B): Below/Above 98.33%, Left/Right 98.10%, Big/Small 95.28%, Tall/Short 96.43%, Wide/Thin 91.34%, Behind/Front 93.64% (vs. SpatialRGPT-8B: 99.17%, 100%, 84.90%, 89.28%, 91.34%, 90.90%).
  • SpatialReasoner-R1 fDPO 4B outperforms InternVL2.5-78B (87.37% vs. 72.29% qualitative; 66.76% vs. 35.25% quantitative).
  • General VL benchmarks (fDPO 8B vs. SpatialRGPT-8B): MME 1667/503 vs. 1667/348; POPE 89.71% vs. 85.50%; SEED-Bench 76.21% vs. 67.00%; AI2D 78.85% vs. 67.42%; SQA-Test 93.85% vs. 81.81%; MMMU 48.11% vs. 41.40%; MMStar 55.43% vs. 43.98%; HallusionBench 51.10% vs. 40.80%.
  • Ablations: α=30%, λ=0.6 are optimal; α=40% or λ=0.8 degrade performance due to over-aggressive segment re-weighting.

Limitations

  • Relies on monocular depth maps as auxiliary input during reward computation (M3CTS and reward scoring)—depth quality may degrade for unusual scenes or domain shifts.
  • Evaluated only on SpatialRGPT-Bench; spatial benchmarks covering 3D scene graphs, egocentric video, or outdoor navigation (e.g., VSI-Bench, 3DSRBench) are not reported.
  • The structured LongCoT output format with explicit <description>/<reasoning> XML-like tags is enforced across all VLMs in M3CTS, which may limit generalization to unconstrained output formats.
  • ChatGPT-4o refuses quantitative distance questions ("/"), leaving a gap in baseline comparison for those tasks.
  • fDPO hyperparameters (α, λ, β) require per-task tuning; no cross-dataset transfer experiments are reported.

Relevance to Vision-Language Models

This work directly targets a known weakness of VLMs—spatial and geometric reasoning from 2D images—through a training-time intervention (fDPO) rather than architectural change or extra sensor modality. The decomposition of LongCoT into functionally distinct segments with segment-specific optimization is a transferable idea applicable to any multi-stage reasoning task in VLMs (e.g., GUI navigation, visual math). The M3CTS data pipeline is a model-agnostic method for bootstrapping high-quality reasoning supervision where annotated data is sparse, connecting to broader efforts in synthetic data generation for VLM alignment. The demonstrated generalization to standard VL benchmarks (POPE, MMStar, HallusionBench) suggests fDPO does not sacrifice general capability for spatial specialization, making it a strong candidate for inclusion in general-purpose VLM fine-tuning pipelines.