Skip to content

SketchMind: A Multi-Agent Cognitive Framework for Assessing Student-Drawn Scientific Sketches

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SketchMind is a four-agent framework that evaluates and improves student-drawn scientific sketches by representing them as Bloom's taxonomy-annotated semantic graphs (Sketch Reasoning Graphs, SRGs). Each agent specializes in rubric parsing, sketch perception via MLLMs, cognitive alignment scoring, and iterative visual feedback with sketch modification. Adding SRG supervision to GPT-4o raises average accuracy from 55.6% to 77.1%; with GPT-4.1, it reaches 90.2%.

Problem

Automated assessment of free-form student science sketches is unsolved: existing approaches treat sketch evaluation as image classification or feed images directly into monolithic vision-language models (e.g., NeRiF/GPT-4V), yielding poor interpretability, no pedagogical alignment to cognitive levels (Bloom's taxonomy), and inconsistent performance across conceptually diverse science tasks.

Method

SketchMind models both a gold-standard reference sketch and a student sketch as Sketch Reasoning Graphs (SRGs): directed graphs G = (V, E, ℓ, λ) where nodes are domain concepts, edges are causal/logical relations, ℓ maps each node to a Bloom level (Remember→Create), and λ captures visual/textual evidence. Four sequential agents operate on these graphs:

  1. Rubric Parser (Agent 1): Parses the rubric and reference sketch to construct the gold-standard SRG G_o, annotating nodes with Bloom levels via a curated verb lexicon; also produces reverse mappings (concept → visual drawing hint) used later by Agent 4.
  2. Perception (Agent 2): Applies an MLLM to the student sketch image to infer the student SRG G_s, assigning semantic roles and Bloom levels to visual elements.
  3. Cognitive Alignment Evaluator (Agent 3): Computes a similarity score S ∈ [0,1] combining graph-edit distance (fGED) and ontology-based node alignment (fOA), with Bloom-level mismatches penalized. Determines the dominant Bloom level achieved by the student and detects cognitive regressions.
  4. Feedback Generator / Sketch Modifier (Agent 4): If S < threshold τ (= 0.75), identifies missing/misaligned nodes/edges, uses reverse mapping ϕ to generate Bloom-aligned visual hints (e.g., overlay arrows with prompts), and iterates until S ≥ τ or Tmax is reached.

Agent 3 is deterministic and model-agnostic; Agents 1, 2, and 4 use the backbone MLLM (kept consistent per experiment). Models tested include GPT-4o, GPT-4.1, GPT-4.1-nano, o3, o4-mini, LLaMA 4 Maverick (400B INT4), and LLaMA 4 Scout (109B INT8).

Key Contributions

  • Sketch Reasoning Graphs (SRGs): A formal, cognitively annotated graph representation embedding domain ontology and Bloom's taxonomy levels for student sketches.
  • SketchMind framework: Four-agent modular pipeline (rubric parsing → sketch perception → cognitive alignment → iterative feedback) grounded in Bloom's taxonomy for transparent, pedagogically aligned sketch assessment.
  • SRG-guided supervision: Demonstrates that providing SRG structure as context to any backbone MLLM yields large, consistent accuracy gains across all tested models and science items.
  • Multi-agent vs. single-agent ablation: Shows modular decomposition improves performance independently of SRG, with further compounding gains when combined.
  • Human-rated iterative sketch modification: Agent 4 generates modified sketches and formative feedback rated 4.1/5 by expert science educators using GPT-4.1.

Results

  • SRG integration gains (GPT-4o): Average accuracy 55.6% → 77.1% (+21.4 pp absolute) across 6 NGSS-aligned science items (3,575 student sketches).
  • Best configuration (GPT-4.1 + SRG + multi-agent): 90.2% average accuracy; individual item gains of +11.0% to +15.4% over GPT-4.1 alone.
  • Open-source gains: LLaMA 4 Maverick +24.9 pp average; LLaMA 4 Scout +21.0 pp average with SRG.
  • Multi-agent vs. single-agent ablation (GPT-4.1, no SRG): 62.9% → 77.4%; with SRG: 82.8% → 90.2%.
  • Multi-agent vs. single-agent ablation (GPT-4o, no SRG): 50.1% → 55.6%; with SRG: 69.5% → 77.1%.
  • Human evaluation of feedback quality (1–5 scale, 4 expert raters, κ = 0.83): GPT-4.1 scores 4.1 average; o3 scores 4.0; GPT-4o scores 2.3; LLaMA 4 Scout scores 2.5.
  • Highest Bloom-level item (H4-1, "Create"): GPT-4.1 multi-agent+SRG reaches 87.2% vs. 57.1% single-agent without SRG.

Limitations

  • Inter-agent coordination is static and predefined; no dynamic planning or adaptive orchestration (e.g., RL-based task decomposition).
  • Evaluation uses only overall proficiency-level accuracy, not fine-grained SRG-level metrics; manually constructed reference SRGs would enable deeper analysis.
  • Dataset is limited to NGSS science items; generalizability to other domains or sketch types (e.g., math, engineering) is undemonstrated.
  • Behavioral data (stroke sequence, eye-tracking) that could enrich SRG modeling is not incorporated.
  • Dataset requires approval before public release, limiting immediate reproducibility of the full pipeline.

Relevance to Vision-Language Models

SketchMind demonstrates a concrete pattern for augmenting MLLMs with structured intermediate representations—SRGs—rather than relying on end-to-end image-to-label inference, achieving +21 pp gains over GPT-4o baseline and showing this holds across both proprietary (GPT-4.1, o3) and open-source (LLaMA 4) vision-language backbones. The multi-agent decomposition isolates the MLLM's visual perception role (Agent 2) from scoring logic (Agent 3, which is deterministic), a design pattern relevant to ongoing debates about grounding and hallucination in VLMs applied to structured domains. For VLM researchers, the result that SRG supervision benefits weaker models disproportionately (LLaMA 4 Scout +21 pp vs. GPT-4.1 +12.8 pp) suggests that semantic scaffolding can substitute for raw model capacity in specialized visual reasoning tasks. The Bloom-level annotation scheme also provides a domain-grounded framework for probing hierarchical visual-semantic understanding in MLLMs.