Skip to content

MindEdit-Bench: Benchmarking Object-Level Counterfactual Spatial Reasoning in VLMs from In-the-Wild Photos

🕒 Published (v1): 2026-07-01 06:19 UTC · Source: Arxiv · link

Why this paper was selected

Counterfactual spatial reasoning benchmark probes genuine VLM scene understanding beyond observation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MindEdit-Bench is a six-task benchmark evaluating whether VLMs can predict the spatial consequences of hypothetically translating or rotating an object in an indoor scene — a capability existing benchmarks do not test. Built from 3-photo smartphone triplets via a feed-forward 3D scene-graph pipeline, it provides 1,003 human-verified questions across 120 private indoor scenes. The best of 15 evaluated VLMs (Gemini-3.1-Pro-Preview) achieves only 34.6% pooled accuracy versus 87.9% human majority vote, a gap of 53.3 percentage points.

Problem

Existing VLM spatial reasoning benchmarks are observational: they ask models to describe relations already present in the input image(s). The two narrow "what-if" paradigms that exist either vary the observer while fixing the scene (viewpoint-change tasks) or evaluate pixel-level generation fidelity (image-editing benchmarks). The object-level counterfactual case — predicting spatial consequences when an object is hypothetically translated or rotated while the observer remains fixed — has not been systematically benchmarked; correct answers are absent from all input images by construction, ruling out 2D visual lookup.

Method

Data collection. 120 indoor scenes are captured as three-photo smartphone triplets (∼810 images total) never appearing in public datasets. A two-stage feed-forward pipeline processes each triplet without dense video, professional scanning, or per-scene optimization:

  • Stage 1 (per-view 2D perception). DA3 (Lin et al., 2025) estimates a metric depth map and camera pose per photo. Gemini 3.1 Pro generates a unified object vocabulary across all three photos. SAM3 (Carion et al., 2025) then produces open-vocabulary instance masks, with within-class NMS to merge over-segmented parts.
  • Stage 2 (cross-view 3D association). Masked pixels are back-projected to 3D point clusters. Clusters are matched across views via a composite score of volumetric overlap, containment, and centroid proximity. Planar surfaces use a separate coplanarity rule. Gravity alignment is recovered by fitting normals to detected horizontal planes (floor, ceiling, table-tops) via PCA and voting.

The resulting object-centric 3D scene graph feeds a question generator that produces six task types across three cognitive groups:

Group Tasks 2D shortcut
I Perception L1 cross-view object relations; L2 camera motion partial
II Perspective Transformation L3a single-view; L3b cross-view none
III Counterfactual Editing L4 spatial editing; L5 cross-view visibility editing none

L4 asks: after A is virtually translated (4 horizontal directions) or pivot-rotated (90°/180°/270° around another object) by a scene-dependent distance, what is A's direction and distance relative to reference C?
L5 asks: which of a candidate list of translations/rotations of object A would cause A to appear (or disappear) in a second camera view, accounting for occlusion?

Answer structure. Each question provides 8–24 multiple-choice options (random-guess baseline 4–13%), structured along spatial axes (direction × distance bin in L3/L4; operation type + fallback in L5), enabling letter-level error diagnosis without chain-of-thought parsing.

Evaluation. 15 VLMs in zero-shot setting (6 API-tier: GPT-5.4, Gemini-3.1-Pro-Preview, Claude-Opus-4-6, Doubao-Seed-2.0-Pro, Qwen3.6-Plus, Kimi-K2.5; 9 open-source: Qwen3-VL-8B/32B, InternVL3-8B, SenseNova-SI-1.1 variants, RoboBrain2.5, VST-7B-SFT, GLM-4.6V-Flash, Qwen3.5-35B-A3B). Human baseline: three independent annotators, majority vote; Krippendorff's \(\alpha = 0.82\).

Key Contributions

  • First benchmark targeting object-level counterfactual spatial reasoning in VLMs, where post-edit configurations are absent from all input images.
  • Fully automated in-the-wild pipeline (DA3 + SAM3 + LLM vocabulary) that produces structured spatial QA from three casual smartphone photos with no professional hardware.
  • Structured 8–24-option answer space enabling letter-level attribution of direction, distance, motion-axis, operation-type, and fallback errors without CoT analysis.
  • 120 privately captured indoor scenes with no public-dataset pretraining overlap, reducing contamination risk.
  • Systematic analysis revealing depth-axis weakness and difficulty-sensitive fallback behavior as structured VLM failure modes.

Results

  • Overall gap: Best VLM (Gemini-3.1-Pro-Preview) achieves 34.6% pooled accuracy vs. 87.9% human majority vote; gap = 53.3 pp. Best non-Gemini model (Qwen3.6-Plus) achieves 31.1%, a 56.8 pp gap.
  • Per-task human accuracy: 80.8%–97.0% (all tasks human-feasible); per-task best-VLM accuracy: 20.0%–58.0%.
  • Human–SOTA gap by task: L1 +39.0 pp, L2 +50.9 pp, L3a +56.7 pp, L3b +65.7 pp (near-chance for all models), L4 +43.9 pp, L5 +46.8 pp.
  • Depth-axis weakness (L4): Pure-depth answers (camera-near/far) 7.4% vs. pure-lateral (left/right) 18.0% vs. mixed diagonals 24.3%; depth–lateral gap = −10.6 pp. Same pattern: L3a −5.5 pp, L2 forward/backward 29.4% vs. left/right 39.0% (−9.6 pp).
  • Error locality: Wrong-direction errors cluster in the adjacent 45° bin at 1.7× random rate; 180° reversals occur at only 0.6× random rate, indicating approximate region sense without full directional commitment.
  • Operation difficulty (L4): Translation 21.8% vs. pivot rotation 13.5% (−8.3 pp).
  • Answer permutation test: Shuffling option letters causes ≤1.4 pp overall shift and ≤6.9 pp per-task shift across three probed models, ruling out letter-position bias as a primary driver.

Limitations

  • Feed-forward pipeline explicitly trades geometric precision for accessibility; residual noise requires human audit (14.4% question rejection rate, 20.9% answer-key correction).
  • Only three photos per scene: sparse multi-view capture limits scene-graph completeness relative to dense-video or scan-based pipelines.
  • 120 scenes is a small absolute dataset size; the benchmark covers only indoor environments.
  • Chain-of-thought prompting results for four models are referenced but not fully reported in the provided text.
  • Private capture protocol limits third-party dataset extension without replicating the collection setup.

Relevance to Vision-Language Models

MindEdit-Bench directly probes a failure mode of frontier VLMs — the inability to maintain and manipulate an internal 3D scene representation — that observational benchmarks structurally cannot detect. The 53 pp human–VLM gap, observed even on models like GPT-5.4, Gemini-3.1-Pro-Preview, and Claude-Opus-4-6, quantifies how far current architectures remain from scene-level spatial reasoning required for embodied and agentic applications. The depth-axis weakness finding provides a mechanistic hypothesis (2D-dominant visual pretraining makes image-plane position more accessible than camera-depth geometry) that has direct implications for training data curation and architecture design. The benchmark's structured answer space and automatic construction pipeline offer a reusable evaluation methodology for measuring progress on counterfactual 3D scene understanding in future VLMs.