Skip to content

Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

🕒 Published (v1): 2026-07-23 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing spatial reasoning benchmarks assume text or coordinate outputs, creating an answer-interface mismatch that prevents image-generation models from being evaluated under the same metrics as text-output VLMs. ProVisE (Protocolized Visual Evaluation) resolves this by constraining image-generation outputs to protocol-defined visual formats and parsing them deterministically into the original benchmark's answer space. A companion benchmark, SpatialGen-Bench, provides 470 samples across 14 subtasks at four capability levels to diagnose where each model family excels or fails.

Problem

Spatial reasoning benchmarks are built for text-output VLMs and require symbolic answers (coordinates \([x_1, y_1, x_2, y_2]\), option labels, textual descriptions). Image-generation models express spatial judgments natively in pixel space — by marking, masking, or drawing — but those responses are incompatible with existing evaluators. The common workaround of using an auxiliary VLM judge introduces its own reliability bias and breaks metric comparability with VLM baselines, leaving generative spatial cognition unmeasured under shared task semantics.

Method

ProVisE framework: Each task is assigned a visual protocol, which pairs (i) a guidance prompt that instructs the model to encode its answer in a predetermined visual form (e.g., instance mark, direction grid, binary region mask, trajectory overlay) with (ii) a deterministic parser that converts the generated image into the structured answer expected by the original metric. The protocol is fixed before any model is evaluated, preventing model-specific tuning.

Protocol routing: If a validated configuration exists, it is reused. Otherwise the Agentic builder operates in three stages: 1. Normalize the benchmark into task-level contracts (input media, answer format, metric). 2. Select a construction route — Reuse (existing protocol), Build (assemble from registered parsing components), or Fallback (constrained auxiliary parser that sees only the generated image, not the source scene or question). 3. Smoke-validate each candidate protocol (generation succeeds, parsing is stable, metric is compatible); a failed protocol may be revised once before rejection.

Accepted protocols are applied unchanged across all target models. Fallback parsers are distinguished from unconstrained post-hoc VLM judging because they receive no source context.

SpatialGen-Bench: 470 manually reviewed samples across 14 subtasks organized into four capability levels — Perception (counting, depth, orientation, size; 145 samples), Understanding (relationship verification, perspective, mental modeling, grounding; 140 samples), Reasoning (multi-hop, state prediction, geometric feasibility; 90 samples), and Interaction (affordance grounding, navigation, trajectory planning; 95 samples). Evaluation metrics are task-native: exact-match accuracy, point-in-mask accuracy, mean mask precision, partial-credit endpoint score, and DFD-success@0.4 (Discrete Fréchet Distance) for trajectory planning.

Key Contributions

  • Identification and formalization of the answer-interface mismatch between generative spatial cognition and existing VLM-centric benchmarks.
  • ProVisE: a benchmark-agnostic framework that converts protocol-constrained visual outputs into metric-compatible predictions, enabling unified scoring of image-generation models and text-output VLMs.
  • Agentic builder: automated three-stage construction and validation of task-specific generation–parser protocols for previously unconfigured benchmarks, validated on six external spatial benchmarks.
  • SpatialGen-Bench: a 470-sample diagnostic benchmark with 14 subtasks, four hierarchical capability levels, and diverse answer forms (discrete, count, mask, state, trajectory).
  • Quantitative characterization of complementary strengths: image-generation models are competitive in pixel-space externalization; text-output VLMs are superior in compositional spatial reasoning.

Results

  • Human ceiling: 87.79% Overall on SpatialGen-Bench.
  • Best text-output VLM: GPT-5.4 at 61.04% Overall (Perception 74.61, Understanding 44.29, Reasoning 56.64, Interaction 69.70).
  • Best image-generation model: GPT Image 2 at 54.49% Overall (Perception 69.58, Understanding 47.14, Reasoning 41.83, Interaction 56.80).
  • Largest human–machine capability gap is in Understanding (38.57 points below the best machine score).
  • Widest task-level gaps occur on Perspective (51.43 pp), Relationship (31.43 pp), and Mental Modeling (28.57 pp).
  • Spatially specialized text-output VLMs (SpaceR, SpaceOm, SpatialRGPT) do not consistently outperform general-purpose models; SpaceR achieves the strongest Relationship score (57.14) but falls short on overall ranking.
  • Universal VLM parsing (replacing task-specific parsers) changes both scores and rankings, confirming that parser choice is a confound in evaluation.
  • Most visual failures stem from incorrect spatial predictions rather than generation, protocol, or parser breakdowns.

Limitations

  • SpatialGen-Bench contains only 470 samples, limiting statistical power for fine-grained subtask comparisons.
  • Fallback tasks rely on a constrained auxiliary parser rather than fully deterministic evaluation, introducing residual judge dependency for a subset of tasks.
  • ProVisE requires an Agentic construction pass for each new benchmark; construction can fail and requires manual fallback review.
  • All models remain substantially below the human reference (best machine: 61.04 vs. 87.79), suggesting the benchmark is unsaturated but also that no strong ceiling effect validates the framework's discriminative range at the top.
  • The paper text provided is truncated; additional stated limitations from later sections may be absent here.

Relevance to Agentic AI / LLM Agents

As agent harnesses increasingly treat image generation as a first-class action — not just text or tool calls — ProVisE's protocol-constrained evaluation approach addresses a real gap: if you can't score a generated image on the same metric as a text answer, you can't compare or audit agent pipelines that mix output modalities. The answer-interface mismatch problem the paper formalizes is exactly the problem you hit when building evaluation infrastructure for agents that route through heterogeneous backends (some returning text, some returning images, some returning structured data). SpatialGen-Bench's four-level capability taxonomy also gives a ready-made diagnostic for choosing which vision backbone to slot into a spatially-demanding agent task — navigation, UI grounding, embodied planning — without hand-rolling your own ablation. The broader methodological contribution (constrain output format → parse deterministically into benchmark answer space) is a reusable pattern for any harness that needs to evaluate models whose native output type doesn't match the benchmark's expected type.