Skip to content

DrawingBench: Evaluating Spatial Reasoning and UI Interaction Capabilities of Large Language Models through Mouse-Based Drawing Tasks

🕒 Published (v1): 2025-12-01 01:18 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DrawingBench is a benchmark that evaluates LLM spatial reasoning and GUI interaction by requiring models to generate sequences of low-level mouse actions (moveTo, click, mouseDown, mouseUp) to draw specified shapes on a canvas — without visual feedback. Evaluating four frontier LLMs on 250 prompts, the framework finds 92.8% perfect-score rate under structured external feedback, and demonstrates that specification clarity matters more than task complexity.

Problem

Existing spatial reasoning benchmarks (CLEVR, SpartQA, RAVEN) use static QA rather than action sequences, and UI interaction benchmarks (WebArena, MiniWoB++) use high-level clicks rather than coordinate-level control. No prior work evaluates the combination of spatial planning, fine-grained GUI manipulation, and multi-turn iterative refinement — three capabilities that real-world agentic systems must coordinate simultaneously.

Method

DrawingBench provides a browser-based 1000×700px canvas with 6 tools, 8 colors, and 3 stroke sizes. LLMs receive a natural language prompt (e.g., "Draw a red triangle inside a blue square") and must output a JSON action sequence with no visual perception — purely text-based reasoning. Evaluation is rule-based across 8 weighted criteria (required tools, required colors, minimum segments, canvas coverage, position constraint, size constraint, syntax validity, coordinate bounds) plus 4 error categories (SYNTAX, COORDINATE, LOGIC, EFFICIENCY). A two-turn multi-turn protocol provides structured, deterministic feedback after Turn 1; Turn 2 is skipped if Turn 1 score ≥ 0.9 (71.2% of tasks). Four models were tested: Claude-4 Sonnet, GPT-4.1, GPT-4.1-mini, and Gemini-2.5 Flash (1,000 total trials).

Key Contributions

  • DrawingBench dataset: 250 prompts across 20 categories and 4 difficulty levels requiring dynamic action-sequence generation, not QA
  • 8-criterion deterministic evaluation: weighted scoring with reproducible, action-level auditability
  • Multi-turn external feedback protocol: rule-based structured feedback (vs. self-critique) as an oversight mechanism
  • Empirical study: 1,000 trials across four frontier LLMs with per-difficulty, per-category, and error-type breakdown
  • Open-source release: framework, prompts, and data for community use

Results

  • Turn 2 average score: 0.954; perfect-score rate (≥0.9): 92.8% (up from 76.8% at Turn 1)
  • Structured feedback yielded +3.2% average improvement and 33% reduction in score variance (std 0.099 → 0.066)
  • Scenes category: +32.8% improvement (0.672 → 1.000) — largest single-category gain
  • Per-model Turn 2 perfect rates: Claude-4 Sonnet 94.4%, GPT-4.1 93.2%, Gemini-2.5 Flash 92.0%, GPT-4.1-mini 90.8%
  • Difficulty paradox: Hard tasks achieved T2 score 0.973 (100% perfect) vs. Medium 0.958 (92.8%), driven by Hard tasks averaging 4.2 explicit constraints vs. 2.8 for Medium
  • Very Hard tasks (long-horizon planning, e.g., 8×8 checkerboard): 60% perfect rate, T2 score 0.869
  • Error reduction: SYNTAX ERROR −100%, COORDINATE ERROR −75%, LOGIC ERROR −67%, EFFICIENCY WARNING −33%
  • Very Hard tasks generated 35% of all Turn 1 errors despite being 8% of the dataset (1.2 errors/task vs. 0.18 for Easy)

Limitations

  • No human performance baseline to contextualize LLM scores
  • Rule-based evaluation cannot assess aesthetic quality
  • Text-only setting excludes vision-based feedback; screenshot-based iterative refinement not tested
  • Two-turn protocol only; diminishing returns observed after Turn 2 (<1% gain) but long-horizon tasks may benefit from more turns
  • Very Hard category (60% perfect) reveals persistent failures in multi-step procedural planning (grid alignment, iteration counting)
  • Small sample sizes for some categories (scenes: n=1, precision test: n=1) limit generalizability of per-category findings

Relevance to Agentic AI / LLM Agents

DrawingBench directly probes agentic capabilities — multi-step action planning, tool-state tracking, and iterative refinement from external feedback — in a controlled, verifiable setting, making it a useful diagnostic for GUI-operating agents. The finding that external rule-based oversight outperforms self-correction (+3.2% vs. self-critique methods like Reflexion) is directly actionable for designing agent feedback loops. The specification clarity result (more explicit constraints → better performance regardless of complexity) informs prompt engineering and task specification for deployed agents. The framework's emphasis on auditability and transparent evaluation addresses a critical gap for trust in autonomous systems, offering a reproducible template applicable beyond drawing tasks to any agent that must execute verifiable action sequences.