ReVSI: Rebuilding Visual Spatial Intelligence Evaluation for Accurate Assessment of VLM 3D Reasoning¶

🕒 Published (v1): 2026-04-27 10:45 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

Rebuilds 3D spatial reasoning evaluation exposing VLM annotation biases; ICML 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ReVSI rebuilds VSI-Bench, a widely used spatial intelligence benchmark for VLMs, by fixing two systematic validity failures: annotation quality drift from 3D reconstruction artifacts, and scene-observability mismatches caused by sparse frame sampling. The rebuilt benchmark with 381 re-annotated scenes and frame-adaptive QA pairs reveals that proprietary models were consistently underestimated while fine-tuned models exhibit high hallucination rates invisible in the prior benchmark.

Problem¶

VSI-Bench and similar spatial-reasoning benchmarks inherit QA ground truth from point-cloud 3D annotations produced for traditional 3D perception tasks. These annotations contain missing objects, mislabeled identities, and corrupted geometry (e.g., wrong room area from Alpha Shape on noisy reconstructions). Simultaneously, benchmarks generate QA pairs under full-scene observability but evaluate models on 16–64 sparsely sampled frames, making many questions unanswerable under actual model inputs. The result is that measured model rankings are systematically distorted.

Method¶

ReVSI rebuilds VSI-Bench in three layers:

Re-annotation. Authors manually re-annotate object labels and 3D bounding boxes across 381 scenes from ScanNetv2, ScanNet++, ARKitScenes, 3RScan, and MultiScan using a custom 3D web interface. Open-vocabulary labels (504 unique vs. 65 in VSI-Bench, 5365 vs. 3185 objects) are assigned by humans with GPT-5.2 used only for verification.

QA regeneration with bias mitigation. All QA pairs are rebuilt with stricter templates and human verification. Counting queries add single-instance and cross-category cumulative templates to break the VSI-Bench "predict-2" bias (53% ground truth was 2). Short-range absolute distance pairs (<1 m) are removed; room size estimation uses manually drawn top-down polygon annotations rather than Alpha Shape; relative direction adds backward-facing templates.

Frame-budgeted evaluation. QA pairs are generated independently for 16/32/64/all-frame settings. Object visibility per frame is computed by rasterizing annotated 3D bounding boxes onto sampled frames using ground-truth camera poses; objects with <5% pixel coverage are manually verified. Questions whose queried objects are not visible under a given budget are excluded from that variant.

Dummy-video controls. For each scene at 16-frame budget, all frames containing any queried object are removed, yielding unanswerable videos with deterministic ground truth (zero count, etc.). These probe whether model predictions are driven by visual evidence or hallucinated priors.

Key Contributions¶

Systematic characterization of two validity pitfalls in existing VSI benchmarks: annotation drift and frame-observability mismatch
Re-annotated benchmark covering 381 scenes, 5365 objects, 504 open-vocabulary labels, with manually verified QA pairs across 7 task types
Frame-adaptive QA variants (16/32/64/all) that guarantee answerability and GT correctness under each frame budget
Dummy-video stress test exposing hallucination and prior-driven prediction in both proprietary and fine-tuned models
Quantified VSI-Bench error rates: 27% wrong / 11% ambiguous on object counting; 25% RAE >10% on room size; systematic physical implausibility in object heights

Results¶

On base (zero-shot) models, open-source models drop up to 40% accuracy on ReVSI vs. VSI-Bench, especially on object counting, relative distance, and relative direction; proprietary models achieve stable or higher performance
Fine-tuned spatial models show significantly smaller gains on ReVSI than on VSI-Bench; scaling post-training data does not consistently improve performance — some fine-tuned models underperform their base models on specific tasks
Dummy-video hallucination test: InternVL3.5 and several fine-tuned models still achieve surprisingly high scores when queried objects are entirely absent from the video, revealing predictions driven by non-visual priors
Proprietary models (Gemini 3 Pro, Qwen3-VL-32B) achieve near-zero hallucination rates on dummy videos; open-source and specialized fine-tuned models reach up to 100% incorrect answers (i.e., they predict nonzero counts with no visual evidence)
At 16-frame sampling, VSI-Bench question answerability drops to ~40% for some tasks; ReVSI eliminates these by construction

Limitations¶

Re-annotation required authors with 3D-dataset expertise and a custom tool; not easily reproducible or extendable by outside researchers
Coverage remains limited to single/small-room indoor scenes (ScanNet, ARKitScenes, etc.); outdoor and large-scale environments are absent
The 16-frame variant excludes Room Size Estimation and Route Planning entirely, narrowing task coverage at common VLM input budgets
Object Appearance Order task is dropped rather than fixed, reducing task breadth compared to VSI-Bench
Evaluation currently restricted to video-based input; image-only or point-cloud-input VLMs are not directly addressable with the current protocol

Relevance to Vision-Language Models¶

ReVSI directly affects how 3D spatial reasoning capability is measured in VLMs, a capability increasingly claimed by frontier multimodal models. The finding that VSI-Bench systematically reverses the proprietary vs. open-source ranking has immediate implications for model selection and post-training decisions in the field. The dummy-video diagnostic provides a controlled method to distinguish genuine spatial grounding from hallucinated category priors, which is a key open question in understanding VLM reliability. The frame-budgeted protocol generalizes to any video-VLM benchmark where sparse sampling creates an observability gap between annotation and inference contexts.