Skip to content

SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketches

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SketchMind is a four-agent framework that evaluates student-drawn scientific sketches by converting both rubric references and student drawings into Bloom's Taxonomy-annotated semantic graphs (Sketch Reasoning Graphs, SRGs), then scoring structural/semantic similarity between them. On 3,575 NGSS-aligned sketches, adding SRG supervision raises GPT-4o's average accuracy from 55.6% to 77.1% (+21.4 pp) and GPT-4.1's from 77.4% to 90.2% (+12.8 pp). The multi-agent decomposition further outperforms a monolithic single-agent baseline across all tested models.

Problem

Automated assessment of free-form student sketches is poorly served by monolithic vision-language models, which treat the task as image classification without interpretability, pedagogical grounding, or cognitive-level alignment. Prior systems (e.g., NeRiF) lack transparent reasoning, fail to adapt feedback across Bloom levels, and cannot iteratively modify sketches to scaffold learning.

Method

SketchMind decomposes assessment into four sequential, specialized agents operating over SRGs defined as \(G = (V, E, \ell, \lambda)\) where \(V\) are ontology-grounded concept nodes, \(E\) directed causal edges, \(\ell: V \to B\) a Bloom-level annotation per node, and \(\lambda\) visual/textual evidence.

  1. Agent 1 (Rubric Parser): Parses rubric and reference sketch → gold-standard SRG \(G_o\) with Bloom-level labels; also generates reverse visual mappings \(\varphi: (v,e) \to \text{VisualHint}\) for downstream use.
  2. Agent 2 (Perception): Applies an MLLM to student sketch \(x\) → infers student SRG \(G_s\) with semantic roles and Bloom levels.
  3. Agent 3 (Cognitive Alignment): Computes similarity score \(S(G_s, G_o) \in [0,1]\) via a weighted combination of graph-edit distance and ontology-based node alignment: $\(S(G_s, G_o) = 1 - \left(\gamma_1 \cdot \frac{f_\text{GED}(G_s, G_o)}{Z} + \gamma_2 \cdot (1 - f_\text{OA}(V_s, V_o))\right)\)$ with Bloom-level mismatches penalized in both terms. Dominant Bloom level is \(\hat{y} = \text{mode}\{\ell(v) \mid v \in V_s \cap V_o \Leftrightarrow S > \tau\}\).
  4. Agent 4 (Feedback + Sketch Modifier): When \(S < \tau\), identifies missing/misaligned nodes \(\Delta^{(t)}\), generates Bloom-aligned textual feedback and visual hints via \(\varphi\), and renders Python-code overlays on the sketch canvas; iterates until \(S \geq \tau\) or \(T_\text{max}\).

Agent 3 is deterministic and model-agnostic; agents 1, 2, 4 use the same MLLM backbone throughout (homogeneous per run). SRGs are cached via a shared SRGBuilder class; similarity threshold \(\tau = 0.75\).

Key Contributions

  • Introduce SRGs: cognitively annotated semantic graphs encoding domain concepts, directed causal relations, and Bloom-level labels extracted from both rubric references and student sketches.
  • Four-agent pipeline for rubric parsing, sketch perception, cognitive alignment scoring, and iterative feedback with visual sketch modification.
  • Bloom's Taxonomy integration as a first-class scoring dimension: Bloom-level mismatch contributes to graph-edit and alignment penalties, and drives scaffolded feedback generation.
  • Empirical demonstration that modular multi-agent orchestration outperforms monolithic single-agent pipelines with and without SRG supervision, across both closed-source (GPT-4o/4.1/o3/o4-mini) and open-source (LLaMA 4 Maverick/Scout) MLLMs.
  • Dataset: 3,575 NGSS-aligned student sketches across six science items; human evaluation with four expert raters (\(\kappa = 0.83\) inter-rater reliability).

Results

  • SRG integration gains (average accuracy):
  • GPT-4o: 55.6% → 77.1% (+21.4 pp)
  • GPT-4.1: 77.4% → 90.2% (+12.8 pp)
  • GPT-4.1-nano: 62.0% → 73.9% (+11.9 pp)
  • O3: 77.2% → 89.0% (+11.8 pp)
  • O4-mini: 73.3% → 80.8% (+7.5 pp)
  • LLaMA 4 Maverick: 47.8% → 72.4% (+24.9 pp)
  • LLaMA 4 Scout: 43.8% → 64.8% (+21.0 pp)
  • Multi-agent vs. single-agent (GPT-4o + SRG): 69.5% → 77.1%; GPT-4.1 + SRG: 82.8% → 90.2%
  • Human expert feedback quality (GPT-4.1): average 4.1/5 vs. 2.3/5 for GPT-4o baseline; rated on Clarity, Conceptual Accuracy, and Instructional Value
  • Largest per-item SRG gain: GPT-4o on item H4-1, +28.8 pp (47.7% → 76.5%)

Limitations

  • Dataset lacks a publicly available large-scale general-purpose benchmark; the study relies on a single dataset (Zhai et al.) from NGSA; generalizability to other curricula or sketch types is unverified.
  • Agent homogeneity constraint (same MLLM per run) prevents mixing stronger/weaker models per role to optimize cost-performance.
  • The iterative sketch modification loop (Agent 4) renders Python overlays locally, limiting deployment in browser-based or cloud-hosted environments without a local Python runtime.
  • Similarity threshold \(\tau\) and weights \(\gamma_1, \gamma_2\) are calibrated on the same dataset used for evaluation; no held-out calibration set is described.
  • Dataset release is pending institutional approval, limiting immediate reproducibility of training/calibration steps.
  • Human evaluation covers only 25% of the dataset (890 samples); full-dataset qualitative validation is absent.

Relevance to Harnesses / Meta-Harnesses

SketchMind is a directly relevant example of a task-specific multi-agent harness: it decomposes a complex, end-to-end evaluation pipeline into sequential specialized agents, each with a defined input/output contract (SRG construction, perception, scoring, feedback), analogous to how meta-harnesses orchestrate specialized subagents over structured intermediate representations. The SRG acts as a shared, structured intermediate artifact passed between agents — a canonical harness pattern where inter-agent communication is mediated by typed data structures rather than raw natural language. The ablation comparing single-agent vs. multi-agent pipelines provides direct empirical evidence for the modularization benefit that harness architectures claim, and the iterative feedback loop (Agent 4 → student → Agent 2 → Agent 3) demonstrates a closed-loop agentic harness pattern relevant to any domain requiring iterative refinement under a structured objective.