Skip to content

SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketches

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SketchMind is a four-agent framework for automatically assessing and improving student-drawn scientific sketches by representing sketches as Sketch Reasoning Graphs (SRGs) annotated with Bloom's Taxonomy cognitive levels. It decomposes the assessment pipeline into rubric parsing, sketch perception, cognitive alignment scoring, and iterative feedback with visual sketch modification. Integrating SRGs with GPT-4.1 achieves 90.2% average accuracy on a 3,575-sketch benchmark, a +12.8% gain over the monolithic baseline.

Problem

Existing automated scientific sketch assessment treats the task as image classification or uses monolithic vision-language models (e.g., GPT-4V/NeRiF), which lack interpretability, pedagogical alignment across Bloom's cognitive levels, and the ability to generate actionable, personalized feedback. There is no mechanism to map visual sketch elements to structured cognitive representations or scaffold iterative student improvement.

Method

SketchMind represents both a reference sketch (from the rubric) and a student sketch as Sketch Reasoning Graphs (SRGs): directed semantic graphs G = (V, E, ℓ, λ) where nodes are domain concepts, edges are causal/logical relations, and ℓ maps each node to a Bloom level (Remember→Create). Four sequential agents operate on these graphs:

  1. Rubric Parser (Agent 1): Parses the rubric and gold-standard sketch to construct a reference SRG Go with Bloom-level annotations, and produces a reverse mapping φ: concept → visual hint.
  2. Perception (Agent 2): Uses an MLLM to extract the student SRG Gs from the raw sketch image, inferring semantic roles and Bloom levels.
  3. Cognitive Alignment Evaluator (Agent 3): Computes a similarity score S(Gs, Go) ∈ [0,1] combining graph-edit distance (fGED) and ontology-based node alignment (fOA), with Bloom-level mismatches penalized. Determines the dominant Bloom level achieved.
  4. Feedback Generator (Agent 4): When S < threshold τ, identifies missing/misaligned nodes, uses the reverse mapping φ to generate Bloom-aligned visual hints (e.g., overlaid arrows with prompts), and renders Python-generated canvas modifications. The loop repeats up to Tmax iterations until S ≥ τ.

Agent 3 is model-agnostic and deterministic; Agents 1, 2, 4 use the same backbone MLLM. Evaluated with GPT-4o/4.1/4.1-nano/o3/o4-mini (OpenAI API) and INT4-quantized LLaMA-4 Maverick/Scout (4×H100).

Key Contributions

  • SRGs: a formally defined cognitively annotated semantic graph representation embedding domain concepts, directed relations, and Bloom's Taxonomy levels for scientific sketches.
  • A four-agent modular pipeline decomposing sketch assessment into rubric parsing, perception, alignment scoring, and iterative feedback with visual sketch modification.
  • Empirical demonstration that SRG integration yields consistent +10–29% accuracy gains across all tested MLLMs on a 3,575-sketch NGSS-aligned dataset.
  • Ablation confirming multi-agent orchestration outperforms single-agent baselines both with and without SRG (e.g., GPT-4.1: 62.9% → 77.4% multi-agent w/o SRG; 82.8% → 90.2% multi-agent w/ SRG).
  • Human evaluation protocol (κ=0.83 inter-rater agreement) rating feedback quality; GPT-4.1-backed SketchMind achieves 4.1/5 vs. 2.3/5 for GPT-4o baseline.

Results

  • GPT-4.1 + SRG (multi-agent): 90.2% average accuracy across 6 items vs. 77.4% without SRG (+12.8%) and 82.8% single-agent w/ SRG.
  • GPT-4o + SRG (multi-agent): 77.1% average vs. 55.6% without SRG (+21.4%); single-item peak gain of +28.8% (Item H4-1).
  • LLaMA-4 Maverick + SRG: 72.4% average vs. 47.8% baseline (+24.9%), with item-level gains up to +29.7%.
  • LLaMA-4 Scout + SRG: 64.8% average vs. 43.8% (+21.0%).
  • Human feedback ratings (Agent 4): GPT-4.1: 4.1/5; o3: 4.0/5; LLaMA-4 Maverick: 3.5/5; GPT-4o: 2.3/5; LLaMA-4 Scout: 2.5/5.
  • Multi-agent vs. single-agent (GPT-4.1, w/o SRG): 77.4% vs. 62.9%; highest-Bloom item H4-1 specifically: 73.1% vs. 57.1%.

Limitations

  • Inter-agent coordination is static and predefined; no dynamic task planning or adaptive agent routing.
  • Evaluation uses holistic proficiency-level prediction accuracy; SRG-level node/edge evaluation against manually curated expert SRGs is not performed.
  • No student behavioral signals (stroke order, eye tracking) are incorporated into SRG construction.
  • Dataset is single-domain (NGSS science sketches); generalizability to other domains or sketch types is undemonstrated.
  • All agents share the same backbone model per run; cross-agent model heterogeneity (e.g., cheap perception + expensive alignment) is unexplored.

Relevance to Agentic AI / LLM Agents

SketchMind is a concrete domain-specific instantiation of the principle that decomposing a complex reasoning task across specialized agents with a shared structured representation (SRGs) consistently outperforms monolithic LLM baselines — directly validating multi-agent modularization as a performance lever, not just a design preference. The ablation isolating multi-agent vs. single-agent contributions is unusually clean for this literature and provides quantitative evidence that task decomposition itself (not just model scale) drives gains. The iterative feedback loop in Agent 4 — using graph-diff to drive targeted visual hint generation and canvas modification until a convergence threshold is met — is an applied example of tool-augmented, goal-conditioned agentic loops. For researchers tracking LLM agents, this paper is a worked case study in grounding agentic coordination in a formal intermediate representation (the SRG) to enable interpretable, controllable multi-step reasoning.