Skip to content

From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP

🕒 Published (v1): 2026-06-25 02:18 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; structural diagnostic separating language priors from true spatial reasoning in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CRISP is a structural-diagnostic benchmark that couples spatial QA with 3D Scene Graph Construction (SGC) to expose whether VLMs genuinely perceive spatial structure or merely exploit language priors. By measuring cross-task consistency between explicit QA answers and implicitly externalized geometry, CRISP reveals a systematic perception-reasoning disconnect: proprietary models have strong latent reasoning but fail to anchor it in their own visual representations, while open-source models lack the multi-hop compositional reasoning to leverage even accurate perception.

Problem

Standard spatial VQA benchmarks allow models to answer correctly via language priors and semantic shortcuts without actual 3D geometric grounding. Existing diagnostic probes (textual rationales, 2D cognitive maps) either suffer from hallucinations themselves or lack metric precision. There is no rigorous way to distinguish "correctly guessed" from "genuinely perceived and reasoned" spatial answers.

Method

CRISP combines two coupled tasks over 1,162 single-view indoor/outdoor scenes (nuScenes + ScanNet++, 1:1 ratio, 9,839 QA pairs):

  1. Spatial QA — seven competency categories (distance, size, direction, counting, ranking, view transformation, logical deduction) in both MCQ and Numerical Answer Question (NAQ) formats. Questions are procedurally generated from a Master 3D Scene Graph using numeric object IDs, preventing semantic label leakage.

  2. 3D Scene Graph Construction (SGC) — given a center object, the model generates a star-topology subgraph predicting node intrinsics (3D dimensions, ego-centric distance) and edge attributes (directional predicates + metric Euclidean distances). Camera intrinsics are withheld deliberately to test whether models use world-knowledge priors for metric recovery.

Three metrics are computed: - QA Score: Accuracy (MCQ) and Mean Relative Accuracy \(\text{MRA} = \frac{1}{10}\sum_{\theta \in C} \mathbf{1}(|\hat{y}-y|/y < 1-\theta)\) for NAQ. - SGC Score: \((\mathcal{S}_{est} + \mathcal{S}_{rel})/2\), where \(\mathcal{S}_{est}\) averages ratio-based size IoU and relative-accuracy distance scores, and \(\mathcal{S}_{rel}\) scores mutually-exclusive directional predicate pairs. - Self-Consistency Score: A deterministic rule-based solver derives QA answers from the model's own SGC output (\(A_{\text{derived}}\)) and checks agreement with its direct QA response (\(A_{\text{QA}}\))—measuring internal coherence, not ground-truth correctness.

An intervention ablation contrasts performance under ground-truth SG guidance vs. self-predicted SGs to decouple perceptual bottlenecks from reasoning deficits.

Key Contributions

  • CRISP benchmark: 1,162 scenes, 9,839 QA pairs, dual-task (QA + SGC) paradigm targeting static geometric grounding.
  • Cross-task consistency metric: Evaluates alignment between linguistic QA output and externalized structural topology—distinguishes true visual reasoning from semantic shortcuts.
  • Intervention-based diagnostic protocol: Dual-intervention experiment (GT SG vs. predicted SG) quantifies the perception-reasoning disconnect in 13 state-of-the-art VLMs.
  • Empirical finding: Proprietary models suffer from utilization failure (good perception, poor anchoring); open-source models suffer from reasoning deficit (shortcuts, no structural grounding).

Results

  • Best proprietary QA: Gemini-2.5-Pro 58.93, GPT-5-Mini 54.15; best open-source QA: Qwen3-VL-8B 55.34.
  • Best proprietary SGC: Gemini-3-Flash 68.79; best open-source SGC: Qwen3-VL-32B 62.45.
  • Best consistency: Gemini-2.5-Pro 57.28; open-source leader Qwen3-VL-8B 52.16.
  • Gemini-2.5-Flash matches Pro on SGC (\(\Delta\)SGC=+0.34) but trails by 11.25 QA points—canonical perception-reasoning disconnect.
  • LLaVA-OV-1.5 surpasses Gemini-2.5-Flash on QA yet has lower consistency, exposing shortcut reliance.
  • Specialized spatial models (VG LLM, Cambrian-S) underperform generalist baselines on QA despite comparable or better SGC scores, suggesting fragmented cross-task alignment from spatial fine-tuning.
  • Blind (text-only) test establishes a ~40 SGC floor; Qwen2.5-VL shows \(\Delta\)QA=+16.31 but only \(\Delta\)SGC=+3.24 with visual input, confirming a Semantic-Geometric Gap where semantic recognition drives QA without geometric localization.
  • All models produce >99% syntactically valid scene graphs in text-only mode, ruling out instruction-following as the source of low SGC scores.
  • "Consistent hallucination" (coherent but geometrically wrong) occurs in ~10–12% of instances across models.

Limitations

  • Restricted to static, single-view images; temporal/video spatial reasoning is excluded by design.
  • Metric estimation from a single image without camera intrinsics is mathematically ill-posed; results reflect world-prior exploitation, not true monocular metric depth.
  • 1,162 scenes—sufficient for diagnostic statistics but small relative to large-scale VQA benchmarks.
  • SGC star-topology subgraphs are a proxy for local topology; full-graph generation complexity is deferred.
  • Thinking-mode suppression for proprietary models introduces a confound: results may understate reasoning capacity of models where extended thinking is the normal inference mode.
  • Dataset limited to two sources (nuScenes, ScanNet++); domain coverage is constrained.

Relevance to Vision-Language Models

CRISP directly challenges the validity of standard VQA-style spatial benchmarks for VLMs by showing that correct answers do not imply correct perception—a finding with broad implications for how spatial capability is reported and optimized. The perception-reasoning disconnect it uncovers reframes the development target: the bottleneck for proprietary VLMs is structural alignment (getting the reasoning engine to consult its own visual representations), not logical capability, while open-source VLMs need stronger multi-hop compositional reasoning over grounded geometry. The SGC generative probe and cross-task consistency metric provide a diagnostic template that could be applied to other implicit VLM representations beyond spatial structure. For researchers tracking VLMs, this work also raises a methodological warning: spatial fine-tuning that optimizes QA accuracy may actively entrench shortcut learning rather than improving genuine 3D cognition.