Skip to content

SPACENUM: Revisiting Spatial Numerical Understanding in VLMs

🕒 Published (v1): 2026-05-22 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SPACENUM is a benchmark evaluating whether VLMs genuinely ground numerical values in spatial meaning, covering two complementary settings: numbers as dynamic transition magnitudes (spatial exploration) and numbers as static coordinate layouts (spatial understanding). Across 18 VLMs, all models perform near random chance, revealing a fundamental gap between producing plausible numbers and truly understanding metric space. The paper also diagnoses failure modes and shows that fine-tuning with a layout-heavy data recipe partially closes the gap and transfers to external benchmarks.

Problem

VLMs deployed in embodied agents must generate explicit numerical outputs (e.g., rotation angles, object coordinates) grounded in metric spatial properties. Prior work treats spatial reasoning qualitatively; it remains unknown whether models truly understand the numerical magnitudes they produce or merely mimic plausible surface patterns. No unified framework existed to probe this bidirectionally—mapping from numbers to visual outcomes and from visual observations to numbers.

Method

SPACENUM defines two orthogonal axes:

Spatial scenarios: - Dynamic transition — parameterized embodied actions (Move F/B, L/R; Rotate U/D, L/R) collected in AI2-THOR with action magnitudes ranging from 0.2–2.4 m (movement) and 10–70° (rotation). - Static layout — 1D/2D/3D coordinate-based cognitive maps built in NVIDIA Isaac Sim with anchor-defined reference frames, desktop- and room-scale scenes.

Task directions: - \(\text{NUM2SPACE}\): given \((o_t, a, n)\), select the correct resulting observation \(o_{t+1}\) from candidates varying only in action magnitude; or given a cognitive map \(M\), select the matching layout image. - \(\text{SPACE2NUM}\): given \((o_t, o_{t+1}, a)\), infer the numerical transition magnitude \(n\); or given a layout image, infer coordinates \(p\) of a target object.

The benchmark contains 3,800 evaluation samples (multiple-choice) plus 77,412 training samples for tuning experiments. Interventions include blind testing (black images), visual simplification (adding anchors, reducing objects), numerical reformatting (natural language, integer scaling, structured abstractions), and fine-tuning with LoRA (rank 8, lr \(10^{-4}\), 3 epochs) and GRPO RL (rank 64, 5 rollouts/prompt, strict vs. graded reward).

Key Contributions

  • A unified bidirectional benchmark (NUM2SPACE / SPACE2NUM) spanning both dynamic and static spatial numerical understanding, with 3,800 evaluation QA pairs and a 77K training set.
  • Systematic evaluation of 18 VLMs (2B–72B) from 6 families, showing near-random performance (best model: 39.8% avg vs. 30.0% random).
  • Structured failure analysis: error proximity (larger models make closer numerical errors without better exact-match), error decomposition (static layout failures are dominated by coupled position-and-size errors rather than isolated attribute errors), and modality asymmetry (dynamic tasks are vision-dependent; static layout tasks partially solvable via language priors).
  • Controlled ablations ruling out surface-form reformulations and visual simplifications as bottlenecks; vision-to-structure abstraction identified as the primary failure.
  • Demonstration that explicit chain-of-thought reasoning yields \(<1\%\) gain, LoRA fine-tuning partially improves performance, and a layout-heavy data recipe (75% layout / 25% transition) is optimal; improvements transfer to OmniSpatial-Motion (+4.5–5.5%) and SAT benchmarks (+8–43%).

Results

  • Overall: Best model (Qwen2.5-VL-72B) achieves 39.8% avg accuracy vs. 30.0% random baseline; several models fall below random.
  • Dynamic transitions: Best performance ~40%; models show no action-type specialization; SPACE2NUM consistently outperforms NUM2SPACE across all action types.
  • Static layouts (1D): Models reach ~60–70% on simpler 1D-map tasks (D/R); performance collapses to near-25% random for 2D and 3D maps.
  • Blind testing: Masking images causes large drops for dynamic transitions but minimal impact on static layout, confirming modality asymmetry.
  • Reasoning (think vs. non-think): Performance differences \(\leq 1\%\) across InternVL3.5 and Qwen3-VL families.
  • Visual abstraction: Replacing layout images with 3D bounding boxes substantially improves SPACE2NUM, confirming vision-to-structure abstraction as the bottleneck.
  • LoRA tuning: Graded RL reward outperforms strict reward (e.g., Layout: +7.60 vs. +6.64 with GRPO); 1D tuning partially transfers to 2D/3D.
  • External transfer (Qwen3-VL-8B): OmniSpatial-Motion +4.5%, SAT-AC +18.9%, SAT-OM +43.5%.

Limitations

  • Evaluation is multiple-choice only; open-ended numerical generation is not assessed, potentially underestimating the difficulty.
  • Static layout scenes use controlled, synthetic environments (Isaac Sim/BlenderKit) that may not reflect photorealistic real-world complexity.
  • Fine-tuning experiments are limited to two model sizes (4B and 8B); large-model tuning (70B+) is not explored.
  • RL with GRPO yields only limited gains, and reward design impact is small—alternative RL formulations are not explored.
  • Transfer experiments cover three external benchmarks; generalization to open-world embodied tasks remains unverified.
  • The benchmark currently lacks continuous/free-form numerical output tasks, which are common in real robot action spaces.

Relevance to Vision-Language Models

SPACENUM directly probes a critical weakness of VLMs deployed in embodied and agentic contexts: the assumption that numerically formatted outputs are metrically grounded. The finding that near-random performance persists even in 72B models—and that chain-of-thought reasoning provides negligible benefit—challenges the belief that scale and reasoning are sufficient for spatial grounding. The identification of vision-to-structure abstraction as the primary bottleneck (rather than numerical surface form or reasoning length) provides a concrete research target for improving VLMs in robotics, navigation, and spatial QA. This work complements VLM spatial benchmarks like SpatialBench and OmniSpatial by focusing specifically on the bidirectional number–space mapping, exposing asymmetries (NUM2SPACE vs. SPACE2NUM, dynamic vs. static) that prior holistic benchmarks conflate.