Skip to content

Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoning

🕒 Published (v1): 2026-06-30 07:33 UTC · Source: Arxiv · link

Why this paper was selected

Tat-Seng Chua; probes overestimate VLM grounding — critical eval result

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Linear probes and training-free steering can systematically overstate what VLMs actually ground in images. On spatial reasoning, a vision-ablation arbiter (replace the image with a gray blank) reveals three regimes probing conflates: grounded (horizontal, vision-dependent), prior (vertical, a language-side default-to-above), and inverted (depth, below-chance because the model reads visual cues with the wrong sign). The finding holds across 14 VLMs and 6 LM families.

Problem

The standard decode-and-steer pipeline — probe a target attribute from activations, then recover it with a training-free intervention — cannot distinguish grounded visual knowledge from non-visual priors or inverted representations. On camera-egocentric spatial reasoning, the most decodable axis (vertical, 94% probe accuracy) is the least grounded, and a below-chance axis (depth) is caused by mis-signed visual deployment rather than absent knowledge. The failure is invisible to probing and steering alone.

Method

  • Task: Camera-Relative-Direction split of ViewSpatial-Bench (egocentric 4-option spatial QA, 1773 items), decomposed into horizontal (left/right), vertical (above/below), and depth (front/back) axes (1047 pure-axis items).
  • Probe: Five-fold CV logistic regression (PCA-50) on the decision-token residual stream, per-axis and per-layer, over a shuffled-label floor.
  • Training-free projection: Amplify per-axis direction over top-k neurons in the L20–24 band with a single global scalar \(\gamma\), no gradients or labels.
  • Vision-ablation arbiter: Re-evaluate any condition with the image replaced by a same-size gray blank (plus four controls: black, noise, patch-scramble, and a real-but-mismatched same-axis image). Classifies axes by the signed real−gray gap: grounded (real \(\gg\) gray \(\to\) chance), inverted (real below chance, gray reverts toward chance), prior (real \(\approx\) gray).
  • Generality battery: 14 VLMs across 6 LM families (Qwen2/2.5/3, Mistral, SmolLM, Gemma), 2B–27B, plus two external benchmarks (What'sUp, 3DSRBench).

Key Contributions

  • Three-regime grounding taxonomy — grounded-correct, grounded-inverted, and prior-driven — that linear decodability and behavioral accuracy each fail to identify; the most decodable and recoverable axis is the least grounded.
  • Vision-ablation arbiter: a cheap, discriminating, one-forward-pass causal control robust to five ablations, argued as a default control for latent-knowledge and steering claims in VLMs.
  • Architecture-generality: across 14 VLMs / 6 LM families, horizontal is grounded, vertical a prior, depth inverted, with depth inversion emerging at scale within families.
  • Task-type boundary: the inversion is camera-egocentric specific — the same models that invert camera-relative depth are grounded-correct on near-field front/behind (What'sUp) and metric closer-to-camera (3DSRBench).
  • Correction-complexity spectrum: the decode\(\neq\)deploy inversion replicates on 7/8 models across 5 families; minimal correction ranges from a training-free rotation (Qwen3-VL-8B: 21% \(\to\) 77%, matching a trained edit) to trained low-rank edits for distributed inversions.

Results

  • Qwen2.5-VL-7B per-axis behavior: Horizontal 74.9% (18-way) / 81.3% (binary); Vertical 26.9% / 58.5%; Depth 16.3% / 30.7% (below chance at \(p<10^{-5}\)).
  • Probe decoding peaks: Horizontal 84.9%, Vertical 94.1%, Depth 77.2% — all well above shuffle floors (55–63%), vertical the highest.
  • Training-free projection lifts vertical: 59% \(\to\) 79% at \(\gamma=2\) (+21), horizontal unchanged.
  • Vision ablation reveals regimes: Horizontal collapses 82% \(\to\) 49% (vision-dependent); vertical identical 59% \(\approx\) 60% (prior); depth rises 31% \(\to\) 45% (inversion caused by looking).
  • The projection gain does not need the image: vertical gains +21 (real) vs +19 (gray blank) — it amplifies a non-visual default-to-above prior (recall(above) \(\approx\) 99%, recall(below) \(\approx\) 40%).
  • Cross-model: vertical decode 87–97% across all capable models, yet behavior is prior or inverted on every one; depth inversion on 7/8 models across 5 families.
  • Scale emergence: depth inversion deepens monotonically within InternVL3 (38% \(\to\) 31% \(\to\) 28%) and Qwen2.5-VL (48% \(\to\) 32%) as LM scale increases.
  • Training-free rotation on Qwen3-VL-8B: depth 21% \(\to\) 77%, matching a trained low-rank edit (+49\(\pm\)8 vs +55\(\pm\)2).

Limitations

  • The study diagnoses one task type (camera-egocentric direction); the taxonomy may not transfer to all spatial or non-spatial VLM capabilities.
  • The correction-complexity spectrum (§6) is only partially described — the lowest-rung correction that re-deploys depth varies per model and is not always training-free.
  • The arbiter classifies well at the axis level but does not resolve mixed/entangled cases (depth resists a clean single-sign flip; best training-free neutralization only reaches \(\approx\)47% vs a theoretical \(\approx\)68% sign-flip).
  • The probe uses logistic regression on PCA-reduced residual-stream activations; more expressive probes might reveal structure the linear probe misses, though that would not address the core decode\(\neq\)deploy claim.
  • No training-based correction or finetuning is explored as a general remedy, only diagnostic edits.

Relevance to Vision-Language Models

This paper is a critical caution for the growing "latent knowledge" and "steering" literature in VLMs: decodability and steering efficacy are not evidence that a representation is visually grounded. The vision-ablation arbiter is a trivial-yet-discriminating control that should become standard practice for any claim of recovered latent visual knowledge. The finding that depth inversion is camera-egocentric specific and scale-emergent also connects to broader questions about how VLMs acquire (or fail to acquire) genuine 3D spatial understanding vs. linguistic heuristics.