Pursuing Minimal Sufficiency in Spatial Reasoning¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MSSR (Minimal Sufficient Spatial Reasoner) is a zero-shot, training-free dual-agent framework that improves 3D spatial reasoning in VLMs by iteratively constructing a Minimal Sufficient Set (MSS) of perceptual information—compact enough to avoid attention dilution yet complete enough to answer the query. A Perception Agent programmatically extracts 3D primitives via visual programming; a Reasoning Agent prunes redundancy and requests missing data in a closed loop. MSSR achieves state-of-the-art on two challenging benchmarks, surpassing proprietary models including o3.

Problem¶

VLMs consistently fail on 3D spatial reasoning due to two compounding bottlenecks: (1) 2D-centric pre-training leaves them geometrically impoverished (no depth, orientation, or layout priors), and (2) naively aggregating all 3D percepts from dense scenes floods the context with weakly relevant details, causing attention dilution and shortcut heuristics that degrade accuracy even when relevant data is present. Prior agentic frameworks address only the first bottleneck (information accumulation) and neglect the second (redundancy management).

Method¶

MSSR formalizes the target as finding \(S^\star \subseteq \mathcal{W}\)—the Minimal Sufficient Set from the full spatial information space \(\mathcal{W}\)—satisfying:

Sufficiency: \(\mathcal{R}^\star(S^\star, q) = a^\star\) (enough to answer \(q\))
Minimality: \(\forall S' \subset S^\star,\ \mathcal{R}^\star(S', q) \neq a^\star\) (no redundant element)

Two agents collaborate in a stateful closed loop:

Perception Agent (PA): Receives a natural-language request \(r\) and current set \(S_n\), generates a Python script that invokes 3D modules (locate, coordinate calibration, reconstruction via VGGT), and merges results into \(S\). A Python environment snapshot is preserved across turns so subsequent calls build on prior computations. A novel SOG (Situated Orientation Grounding) module reframes orientation estimation as a visual multi-choice selection task: candidate 3D direction vectors are overlaid on a Situated View (original image) and a Canonical View (elevated synthetic render), and a VLM selects among them via a coarse-to-fine strategy—halving the angular search space each pass.
Reasoning Agent (RA): Receives \(S_n\), formulates a high-level reasoning plan, prunes each element not causally linked to the plan to produce \(S_{n+1}\), then either issues <Request> (targeted directive back to PA for missing information) or <Decide> (discards all prior context and reasons exclusively over the pruned MSS via CoT). Both agents operate zero-shot with no in-context learning examples.

Key Contributions¶

Formal definition of 3D spatial reasoning as MSS construction with dual criteria (sufficiency + minimality)
Dual-agent closed-loop framework (MSSR) that interleaves stateful programmatic perception with plan-guided pruning
SOG module: reframes language-conditioned orientation grounding as a coarse-to-fine visual selection over rendered 3D candidate vectors, handling both intrinsic object orientation and situation-dependent directions
Zero-shot, training-free design—no 3D instruction dataset or fine-tuning required
Interpretable reasoning traces suitable as supervision data for future 3D-aware models

Results¶

MMSI-Bench overall: 49.5% vs. o3 41.0% (+8.5 pp), Qwen3-VL-8B 31.1% (+60% relative), VLM-3R 32.0%, VADAR 26.1%; +19.2 pp over GPT-4o backbone (30.3%)
ViewSpatial-Bench overall: 51.8% vs. o3 51.1%, Gemini 2.5 Pro 43.0%, InternVL3-14B 40.3%; +16.8 pp over GPT-4o backbone (35.0%)
Ablation (MMSI-Bench / ViewSpatial-Bench):
Full MSSR: 49.5% / 51.8%
Only PA (no pruning): 37.1% / 32.5%
Only RA (no 3D toolbox): 31.1% / 35.3%
w/o SOG: 46.9% / 43.2%
w/o Iteration (single-pass): 47.2% / 48.8%
Conciseness ablation (sufficiency-normalized sets): accuracy peaks at the minimal set size and degrades with both over-inclusion and under-inclusion

Limitations¶

Backbone requires GPT-4o (proprietary); zero-shot capability is backbone-dependent
Multi-round PA↔RA iteration incurs latency; the paper does not report per-query wall-clock cost
SOG is not designed for sub-degree angular precision; adequate for current benchmarks but may fail on fine-grained tasks
Foundational reconstruction depends on VGGT; robustness degrades with sparse or textureless scenes
Paper text is truncated before full discussion of limitations, failure cases, and generalization bounds

Relevance to Harnesses / Meta-Harnesses¶

MSSR is a textbook dual-agent harness: the Perception Agent is a programmatic tool-execution layer (visual programming backbone with stateful Python environment persistence), while the Reasoning Agent acts as a meta-controller that orchestrates perception calls, prunes the shared state, and decides loop termination—exactly the harness-within-harness architecture central to this research thread. The key design insight—that a meta-harness must actively manage context quality, not just accumulate tool outputs—directly challenges the dominant "gather-and-reason" pattern and proposes a "gather-prune-request" loop as a more principled alternative. The state-preserving Python snapshot mechanism is a concrete harness engineering contribution: it avoids redundant re-computation across loop iterations, a problem relevant to any multi-step agentic harness. For researchers building meta-harnesses, MSSR demonstrates that explicit sufficiency and minimality criteria can be operationalized as loop-termination conditions, providing a template for cost-bounded, quality-gated agentic pipelines.