Skip to content

GeoBench: Rethinking Multimodal Geometric Problem-Solving via Hierarchical Evaluation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; GeoBench: hierarchical geometric reasoning evaluation for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GeoBench is a 1,021-sample hierarchical benchmark for evaluating geometric reasoning in VLMs across four progressive capability levels — Visual Perception, Goal-Oriented Planning, Rigorous Theorem Application, and Self-Reflective Backtracking — instantiated as six formally verified tasks. Problems are generated via TrustGeoGen's symbolic proof engine, eliminating test-set contamination from textbooks. Experiments show performance degrades sharply with task complexity, and Chain-of-Thought prompting surprisingly harms self-reflective backtracking.

Problem

Existing geometric reasoning benchmarks for VLMs suffer from three compounding flaws: (1) sourcing problems from public textbooks creates data contamination risk, enabling memorization over genuine reasoning; (2) evaluation reduces to final-answer accuracy, ignoring the intermediate theorem-chaining and proof steps that define geometric rigor; and (3) there is no diagnostic granularity to distinguish whether model failures originate from weak spatial perception, incorrect theorem retrieval, or inability to correct flawed reasoning paths.

Method

GeoBench uses TrustGeoGen — a rule-driven formal engine — to generate geometry problems with symbolic reasoning graphs \(G = (P, S, R, \mapsto)\), where \(P\) is the premise set, \(S\) the derived state collection, and \(R\) the deductive rule set. Each derivation is expressed as \(S_r \xrightarrow{r} s'\), and goal-oriented reasoning paths are extracted via backward chaining from target state \(s_t\).

From these verified graphs, six multiple-choice tasks are constructed across four van Hiele-inspired levels: - Level 1 – Visual Perception: Numerical Perception (extract numeric attributes from diagram; distractors are value-altered or nonexistent-point variants) and Structural Perception (extract geometric relations; distractors are logical negations of valid premises). - Level 2 – Goal-Oriented Planning: Irrelevant Premise Filtering (identify \(p \in P \setminus S_0\) — premises unused in the goal-oriented path; requires \(|S_0| \geq 3\)) and Sub-Goal Decomposition (identify \(S_{r_t} \setminus P\) — the intermediate states immediately preceding the target conclusion via backward chaining). - Level 3 – Rigorous Theorem Application: Theorem Selection (identify the one rule \(r_\text{correct} \notin R_\text{used}\) among three genuinely used rules; requires \(|R_\text{used}| \geq 3\)). - Level 4 – Self-Reflective Backtracking: Faulty Branch Localization (given a wrong reasoning path \(P_\text{wrong}\), identify step\(_0\) — the initial divergence point \(P_\text{faulty} := P_\text{wrong} \setminus P\)).

All tasks use 4-choice MCQ format with principled distractor construction. GeoBench covers 76 constructions, 42 deduction rules, and 40 base scenarios; 285 problems are numerical and 736 are proof-based.

Key Contributions

  • Hierarchical benchmark: Four-level framework (1,021 samples, six tasks) enabling sub-skill diagnosis rather than aggregate answer accuracy.
  • Contamination-free generation: All problems synthetically produced by TrustGeoGen with formal symbolic verification, confirmed OOD relative to GeoQA via t-SNE (disjoint image and solution clusters; GeoBench solutions reach thousand-token scale vs. GeoQA's max 189 tokens).
  • Bottleneck identification: Sub-goal decomposition and irrelevant premise filtering are empirically the most predictive sub-skills for end-to-end problem-solving accuracy.
  • CoT failure discovery: Chain-of-Thought prompting degrades performance on Faulty Branch Localization tasks, suggesting injected reasoning steps can mislead models away from error detection.

Results

  • Difficulty calibration: Gemini-2.5-pro scores 75.00% on OlympiadBench-Geo but only 49.58% on GeoBench-solving; GPT-4o scores 13.39% vs. 22.08% respectively — placing GeoBench between Olympiad and middle-school difficulty.
  • Best general MLLM: Qwen2.5-VL-72b achieves 85.71% (NP), 40.74% (SP), 38.50% (IPF), 77.00% (SD), 47.42% (TS), 22.45% (FBL) — performance collapses at Level 4.
  • Human baselines: 100% on NP/SP/SD, 77.50% on IPF, 56.67% on TS, 52.94% on FBL — demonstrating Level 3–4 tasks are non-trivial even for humans.
  • Most models near or below random (25%) on Structural Perception: e.g., LLaVA-OV-7b scores 22.96% (−2.82%), Qwen2-VL-7b 18.52% (−7.26%), Molmo 12.59% (−13.19%).
  • General trend: All general MLLMs show monotonically declining accuracy from Level 1 to Level 4; reasoning models (OpenAI-o3 cited but full table truncated) are the strongest performers overall.

Limitations

  • Benchmark size is modest (1,021 samples) relative to some prior benchmarks (e.g., MathVista: 5,487).
  • Coverage is bounded by TrustGeoGen's generative vocabulary (76 constructions, 42 rules, 40 base scenes) — exotic geometric configurations outside this repertoire are excluded.
  • All problems are in English only, limiting multilingual evaluation.
  • Tasks are exclusively multiple-choice (4-way), preventing assessment of open-ended proof generation.
  • The faulty branch localization task requires a wrong solution path as input, which may not reflect naturalistic model failure modes.
  • The paper text is truncated; full results for QvQ-72b and reasoning models (OpenAI-o3) are not available in the provided excerpt.

Relevance to Vision-Language Models

GeoBench directly stress-tests the geometric reasoning pipeline of VLMs beyond surface-level answer accuracy, revealing that even frontier models like Qwen2.5-VL-72b score near random on structural perception and sub-25% on faulty branch localization — exposing systematic gaps in spatial understanding and self-correction that aggregate benchmarks obscure. The finding that CoT prompting hurts self-reflective tasks is directly actionable for VLM prompting strategies in mathematical domains. For researchers tracking VLMs, this benchmark establishes a reproducible, contamination-resistant diagnostic suite that separates perceptual from planning from theorem-retrieval from metacognitive failures — a finer-grained lens than any prior geometric benchmark. The correlation analysis between sub-tasks and end-to-end solving accuracy provides a principled roadmap for where to invest in architectural or training improvements.