ORIGAMISPACE: Benchmarking Multimodal LLMs in Multi-Step Spatial Reasoning with Mathematical Constraints

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ORIGAMISPACE is a benchmark of 350 origami instances designed to evaluate multimodal LLMs on multi-step spatial reasoning under rigorous mathematical constraints (Kawasaki's theorem, Huzita-Hatori axioms). It exposes large gaps between current MLLMs and human expert performance across four tasks spanning pattern prediction, step ordering, spatial relationship prediction, and end-to-end CP code generation. Environmental and reinforcement learning paradigms using an interactive origami compiler significantly outperform one-shot in-context generation.

Problem

Existing spatial reasoning benchmarks (CLEVR, NLVR2, LEGO-Puzzles) focus on static scenes, single-step reasoning, or multi-step processes without precise geometric constraints or environment interaction. No prior benchmark evaluates MLLMs on continuous, constraint-governed spatial transformations requiring both multi-step reasoning and adherence to formal mathematical rules.

Method

The authors curate 350 origami data instances, each containing: a crease pattern (CP) diagram (parseable as structured code), a compiled flat pattern (computed by an enhanced origami compiler), a folded shape image, and step-by-step folding process images. They enhance an existing origami compiler to (1) mark crease positions, (2) compute paper stacking order, (3) expose an interactive API for MLLM calls, and (4) return four typed errors (CP Code Syntax Error, Geometrically Impossible Fold, Paper Self-Intersection, Ambiguous Folding State).

Four tasks are defined: Pattern Prediction (MCQ: CP diagram → shape name, 350 Qs); Multi-step Spatial Reasoning (MCQ: reorder shuffled folding-step images, 250 Qs, avg. 7.5 steps); Spatial Relationship Prediction (MCQ: spatial pose, layering, and geometric change after folding, 900 Qs); End-to-End CP Code Generation (120 prompts, evaluated on four dimensions: Topological Structure Similarity via graph metrics, Geometric Similarity via bidirectional Hausdorff distance, Constraint Satisfaction via Maekawa/Kawasaki theorem checking, and Final Folded State comparison). The code generation task is explored under three paradigms: in-context learning, environmental learning (iterative MLLM↔compiler interaction, up to 10 rounds), and reinforcement learning (PPO-based TRICO on Qwen2.5-VL-32B with 471 training instances; rewards tied to compilation success and total quality score).

Key Contributions

• ORIGAMISPACE dataset: 350 verified origami instances with CP diagrams, compiled flat patterns, folding process images, and folded shape images; first benchmark coupling multi-step spatial reasoning with formal mathematical constraints.
• Enhanced origami compiler with crease-position tracking, stacking-order computation, MLLM-callable API, and structured error feedback (four error categories).
• Four evaluation tasks totaling 1,500 MCQs and 120 code generation prompts, with a four-dimensional weighted scoring framework for generated CP code.
• Empirical evaluation of 13 MLLMs (open- and closed-source) plus human baselines; exploration of environmental and RL training for constraint-governed code generation.

Results

• Best open-source model (InternVL2.5-78B / Qwen2.5-VL-72B) achieves ~36–39% on Pattern Prediction vs. 51% (common human) and 98% (expert); best closed-source (Gemini 2.5-pro) reaches ~43%.
• Multi-step Spatial Reasoning: Gemini 2.5-pro 53.45%, GPT-4o 51.81%; human expert 100%, common human 88.52%. Worst models near random (25%).
• Spatial Relationship Prediction (Geometric Change sub-task): all MLLMs cluster 24–32%, even top models well below human expert (85–92%).
• CP Code Generation (in-context, GPT-4o): compilation pass rate 95.03%, total score 28.56/100.
• Environmental learning (GPT-4o): CPR 100%, total score 66.92/100; Gemini 2.5-pro total 65.89/100.
• RL-trained Qwen2.5-VL-32B: CPR 91.03%, total score 45.17/100 — surpasses Qwen2.5-VL-72B under environmental learning (43.81) on total score.
• Constraint Satisfaction under environmental learning remains challenging: best score 56.99% (Gemini 2.5-pro).
• Performance saturates after 8–10 interaction rounds in environmental learning.

Limitations

• Dataset is small (350 instances, 471 training); diversity of origami types and complexity is bounded by manual collection and verification effort.
• CP code generation evaluation depends on the correctness of the enhanced compiler; compiler errors or ambiguities (AFS errors) can affect ground-truth reliability.
• RL training is demonstrated only on Qwen2.5-VL-32B; generalization to other architectures is untested.
• Environmental learning performance saturates, suggesting the benchmark alone cannot overcome fundamental model capability ceilings.
• Human baseline for code generation task is not reported, leaving the upper bound for Task 4 undefined.

Relevance to Vision-Language Models

ORIGAMISPACE directly stresses a capability gap in MLLMs that standard VQA and scene-understanding benchmarks obscure: the ability to track geometric state across sequential spatial transformations under formal mathematical invariants, which is critical for grounded physical reasoning. The benchmark's interactive compiler environment and RL training paradigm offer a concrete framework for studying how VLMs can internalize domain-specific constraints through environment feedback — directly informing work on tool-augmented and agent-style VLMs. The finding that even frontier models (GPT-4o, Gemini 2.5-pro) score below 60% on constraint satisfaction highlights a structural weakness in current multimodal architectures for rule-governed spatial tasks. This is particularly relevant for VLM research targeting robotics, embodied agents, and CAD/engineering applications where spatial constraint adherence is non-negotiable.