Skip to content

Pursuing Minimal Sufficiency in Spatial Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; minimal sufficiency framework for spatial reasoning bottlenecks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs fail at 3D spatial reasoning because they lack geometric priors and because flooding them with redundant 3D percepts degrades reasoning. MSSR, a zero-shot dual-agent framework, addresses this by iteratively constructing a Minimal Sufficient Set (MSS)—the most compact set of spatial facts needed to answer a query—using a Perception Agent (PA) for tool-based 3D extraction and a Reasoning Agent (RA) for pruning and targeted re-querying. MSSR achieves state-of-the-art on both MMSI-Bench (49.5%) and ViewSpatial-Bench (51.8%), surpassing o3 and all open-source VLMs.

Problem

Modern VLMs struggle with 3D spatial reasoning due to two compounding bottlenecks: (1) inadequate 3D geometric understanding from 2D-centric pre-training, and (2) context overload—naively aggregating all available 3D percepts introduces redundant information that causes attention dilution and shortcut heuristics, actively degrading accuracy. Prior agentic approaches focus on information accumulation without addressing the redundancy problem.

Method

MSSR formalizes the target as a Minimal Sufficient Set \(S^\star \subseteq W\), where \(W\) is all derivable scene information. \(S^\star\) satisfies:

  1. Sufficiency: \(R^\star(S^\star, q) = a^\star\) — an oracle reasoner can correctly answer query \(q\) from \(S^\star\).
  2. Minimality: \(\forall S' \subset S^\star,\ R^\star(S', q) \neq a^\star\) — no proper subset suffices.

The framework approximates \(S^\star\) through a closed-loop dual-agent iteration:

  • Perception Agent (PA): Uses visual programming (Python codegen) to invoke a toolbox of specialist modules against multi-view images \(\{I_1, \ldots, I_M\}\). Maintains a persistent execution snapshot across turns to avoid redundant computation. Key novel module: Situated Orientation Grounding (SOG), which reframes language-conditioned 3D direction estimation as a visual multi-choice task using a coarse-to-fine selection over rendered candidate vectors on both original and synthetic canonical views.
  • Reasoning Agent (RA): Receives the current set \(S_n\), formulates a reasoning plan, prunes items not causally linked to that plan to produce \(S_{n+1}\), then either issues a <Request> back to the PA for missing information or triggers <Decide> to answer exclusively from the curated \(S_{n+1}\) via Chain-of-Thought. Both agents operate zero-shot.

Key Contributions

  • Formal definition of 3D spatial reasoning as MSS construction, with sufficiency and minimality as joint optimization objectives.
  • MSSR dual-agent closed-loop framework: PA for visual-programming-based 3D perception, RA for plan-guided pruning and targeted re-querying, with stateful cross-turn execution.
  • SOG module: coarse-to-fine, multi-view visual prompting for language-grounded 3D direction extraction without any 3D supervision or fine-tuning.
  • State-of-the-art results on MMSI-Bench and ViewSpatial-Bench, plus interpretable reasoning traces suitable as training data.

Results

  • MMSI-Bench overall: MSSR 49.5% vs. o3 41.0% (+8.5 pp), GPT-4o 30.3% (+19.2 pp), best open-source Qwen3-VL-8B 31.1% (+18.4 pp), best 3D-VLM VLM-3R 32.0%.
  • ViewSpatial-Bench overall: MSSR 51.8% vs. GPT-4o 35.0% (+16.8 pp), Gemini 2.5 Pro 43.0%, best open-source InternVL2.5-8B 43.2%.
  • Ablation (MMSI-Bench / ViewSpatial-Bench):
  • Full MSSR: 49.5% / 51.8%
  • Only PA (no pruning): 37.1% / 32.5% — confirms redundancy hurts
  • Only RA (no 3D tooling): 31.1% / 35.3% — confirms 3D perception gap
  • w/o SOG: 46.9% / 43.2%
  • w/o iteration: 47.2% / 48.8%
  • Conciseness effect: Accuracy peaks at the smallest sufficient set size (~5.9 items) and degrades as set size increases to ~17 items, directly demonstrating that minimality is load-bearing.

Limitations

  • Depends on external expert models (VGGT for reconstruction, Grounding DINO/SAM for localization) — errors compound upstream.
  • SOG is limited to ground-plane orientations and coarse angular precision; sub-degree accuracy is explicitly out of scope.
  • Latency is uncharacterized; multiple PA/RA iterations plus 3D reconstruction make real-time use unlikely.
  • Evaluated on two benchmarks; generalization to dynamic or outdoor scenes with sparse views is untested.
  • Zero-shot design avoids fine-tuning but also cannot adapt to domain-specific spatial vocabularies.

Relevance to Vision-Language Models

MSSR directly diagnoses and addresses a known failure mode of VLMs—the inability to perform reliable 3D spatial reasoning—without any fine-tuning, making it applicable to any capable backbone. The result that redundant context actively degrades VLM reasoning is a sharp empirical finding with implications for how context is constructed in any tool-augmented VLM pipeline. The SOG module and the MSS construction loop are practical building blocks for VLM systems that must ground language in 3D environments (robotics, embodied agents, AR/VR). The interpretable reasoning traces also open a pathway for generating high-quality synthetic 3D reasoning data to train future spatially-aware VLMs.