Skip to content

Towards Physics-informed Spatial Intelligence with Human Priors: An Autonomous Driving Pilot Study

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current VQA-based evaluation of visual-spatial intelligence (VSI) conflates linguistic ability with genuine spatial reasoning. This paper introduces the Spatial Intelligence Grid (SIG), a structured grid-based representation encoding object layouts and physics-grounded priors, along with SIGBench—a 1.4K-frame autonomous driving benchmark annotated with SIG labels and human gaze traces—to disentangle spatial capability from language priors in MLLMs.

Problem

VQA-style VSI benchmarks require text outputs, which allow models to exploit linguistic shortcuts and conflate failures of visual grounding with spatial reasoning errors. No existing benchmark provides a non-textual, compositional spatial representation that encodes geometry and inter-object relations, or separates intrinsic VSI from language priors in foundation model evaluation.

Method

SIG representation: A 10×10 bird's-eye-view grid encoding object positions (vehicles, traffic signs, lights, ego-vehicle) as JSON coordinates. From SIG, two derived representations are extracted: a directed Spatial Relation Graph (SRG) encoding pairwise direction+distance, and a Spatial Relation Paragraph (SRP) describing relations in natural language.

Evaluation metrics (three novel): - Multi-Level Spatial Matching (MLSM): Bipartite matching between predicted and GT object positions across hierarchical levels (type → type+order → type+order+color), computing P/R/F1/AssA over multiple distance thresholds α; O(n³). - Spatial Relation Graph Similarity (SRGS): Graph edit distance (node/edge substitution, insertion, deletion costs) between predicted and GT SRGs, normalized to [0,1]; O(n³). - Semantic Relational Distance (SRD): Cyclic (directional) and linear (proximal) discrete-scale MAE/MSE/Acc over preposition predictions; O(n).

Human-like SIG: Gaze/saliency maps are projected from image space to grid space via homographic transformation (SVD-solved H matrix), then used as per-object attention weights to produce attention-aware SRGS and SRD metrics that penalize errors on highly attended objects more severely.

ICL experiments: 3-shot in-context learning using SIG annotations (ICL-SIG) vs. multiple-choice VQA annotations (ICL-MC) on GPT-4o and Gemini-2.5-Pro, with cross-domain transfer to MS COCO and ARKitScenes subsets.

Key Contributions

  • SIG: a grid-based, physics-informed VSI representation that encodes spatial structure as a complementary channel to text.
  • Three graph-theoretic evaluation metrics (MLSM, SRGS, SRD) that operate directly on SIG topology and run sub-millisecond per frame.
  • Human-like SIG via homographic projection of gaze attention maps, enabling attention-weighted evaluation metrics.
  • SIGBench: 1,423 driving frames annotated with ground-truth SIG, human-like SIG, SRP, and gaze maps; five task clusters (SIGC, SRPF, human-like SIGC/SRPF, gaze prediction).
  • Empirical demonstration that SIG-based ICL outperforms VQA-based ICL for VSI enhancement in frontier MLLMs.

Results

Zero-shot on SIGBench (Tab. 2): - Human-level MLSM F1: 0.938; best MLLM (Gemini-2.5-Pro): 0.507—a large gap. - Gemini-2.5-Pro best on MLSM and SRD (Directional); GPT-4o best on SRGS (S=0.441). - Human SRD (Directional) Acc: 0.901; best MLLM: 0.331 (GPT-4o).

3-shot ICL on SIGBench-tiny (Tab. 3): - ICL-SIG improves nearly every metric over zero-shot for both GPT-4o and Gemini-2.5-Pro. - ICL-SIG vs. ICL-MC: ICL-MC causes regressions (e.g., Gemini-2.5-Pro SRGS-S drops from 0.224 to 0.185 with ICL-MC; recovers to 0.305 with ICL-SIG).

Cross-domain (Tab. 5, GPT-4o): - SIG-COCO: ICL-SIG raises MLSM F1 from 0.771→0.849, SRD (Proximal) Acc from 0.230→0.643. - SIG-ARKitScenes: MLSM F1 0.727→0.823, SRD (Proximal) Acc 0.233→0.800.

Human-like VSI (Tab. 6): Human-SRGS and human-SRD rankings mirror grid-based results; InternVL2.5-26B and Qwen-VL-2.5-7B lead gaze prediction (PCC 0.951, 0.950).

Runtime: All three metrics run under 1 ms per frame on a CPU for up to 22 objects.

Limitations

  • SIGBench is single-frame only; temporal tracking and dynamic object relations across frames are not evaluated.
  • SIG-based fine-tuning and reinforcement learning experiments are absent; only ICL is explored.
  • Human gaze annotation uses a camera-FOV-derived circular approximation rather than actual eye-tracking data.
  • Cross-domain benchmarks (SIG-COCO, SIG-ARKitScenes) are proof-of-concept subsets, not full evaluations.
  • SRGS and MLSM are both O(n³), which may not scale to dense scenes despite sub-millisecond empirical latency in current test ranges.

Relevance to Vision-Language Models

SIG directly challenges the VQA paradigm central to most MLLM spatial evaluation, showing that non-textual structured representations can both expose and improve spatial reasoning that text-based prompts mask. The finding that 3-shot SIG-based ICL consistently outperforms VQA-based ICL—even with random sample selection—has immediate implications for how future spatial VLM benchmarks should be designed and what modalities beyond text should be included in MLLM training signals. The human-like SIG component introduces a gaze-driven evaluation axis that connects VLM spatial reasoning to cognitive and driver-attention models, a relevant direction as VLMs are increasingly deployed in embodied and driving agents.