Skip to content

MAGMA-Edu: Multi-Agent Generative Multimodal Framework for Text-Diagram Educational Question Generation

🕒 Published (v1): 2025-11-24 03:13 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAGMA-Edu is a training-free multi-agent framework that generates paired text–diagram educational problems through a two-stage Generate–Validate–Reflect loop, using executable code as an intermediate representation for geometric diagram synthesis. It outperforms GPT-4o by +35.3 pp on average textual metrics and +72 pp on image–text consistency (ITC). The framework treats multimodal educational content generation as a closed-loop self-supervised optimization problem rather than a single-pass inference task.

Problem

Current MLLMs fail at educational multimodal generation due to two gaps: (1) semantic misalignment between generated diagrams and problem text (mislabeled geometry, distorted spatial relations), and (2) mathematical hallucination in single-pass generation that produces logically invalid questions. No prior method jointly optimizes text quality and diagram fidelity under explicit pedagogical constraints.

Method

MAGMA-Edu orchestrates two sequential stages, each implementing a Generate–Validate–Reflect (GVR) loop:

Stage 1 — Text refinement: Three agents collaborate iteratively. The Text Generator produces a structured JSON problem draft \(T^{(i)}\). The Text Validator scores it across six dimensions (UO, LR, QF, AA, CA, IDQ). The Text Reflector synthesizes corrective feedback \(\Delta T^{(i)}\) and triggers: $\(T^{(i+1)} = F_\text{text}\!\left(k,\, s,\, r,\, \Delta T^{(i)}\right)\)$ until \(Q_\text{text}(T^{(i)}) \geq \tau_\text{text}\) or \(i \geq I_\text{max}\) (default \(I_\text{max}=5\)).

Stage 2 — Programmatic diagram generation: Given verified text \(T^*\), four agents produce a diagram via executable Python/Matplotlib code. The Code Generator emits drawing code; the Code Executor renders it; the Image Validator assesses \(\{Q_\text{syntax}, Q_\text{visual}, Q_\text{align}\}\) via OCR-based multimodal reasoning; the Image Reflector issues \(\Delta G^{(i)}\): $\(G^{(i+1)} = F_\text{vision}\!\left(T^*,\, r,\, \Delta G^{(i)}\right)\)$

The overall quality objective jointly optimizes: $\(\Phi(T, G) = \alpha \cdot Q_\text{text}(T) + \beta \cdot Q_\text{image}(G) + \gamma \cdot \text{Consistency}(T, G)\)$

Parameters \(\Theta\) (agent prompts, iteration thresholds, evaluation weights) are tuned heuristically, not via gradient descent.

Key Contributions

  • Training-free multi-agent framework (MAGMA-Edu) for joint text–diagram math problem generation, requiring no task-specific fine-tuning.
  • Two-stage GVR pipeline that decomposes the text–image alignment problem into sequential, verifiable optimization loops.
  • Executable code as an interpretable intermediate representation enforcing geometric constraints and reproducibility.
  • A multimodal K–12 mathematics benchmark of 390 problems spanning 78 knowledge points across plane geometry, analytic geometry, solid geometry, trigonometry, and function graphs.

Results

  • vs. GPT-4o (single-agent baseline): Avg-Text 92.31 vs. 57.01 (+35.3 pp); ITC 85.24 vs. 13.20 (+72.0 pp).
  • vs. Nano-Banana (Gemini 2.5 Flash Img): Avg-Text 95.57 vs. 94.77 (+0.80 pp); ITC 97.05 vs. 15.90 (+81.2 pp).
  • Best overall configuration (MAGMA-Edu + Gemini 2.5 Pro): Avg-Text 96.20, ITC 99.12.
  • Ablation — Stage 2 alone drives virtually all ITC gain; Stage 1 alone contributes +0 pp ITC but +35 pp Avg-Text over GPT-4o baseline.
  • Pure LLMs (GPT-5, DeepSeek-R1, Gemini 2.5 Pro) score ITC = 0 without Stage 2; MAGMA-Edu wrapper raises them to 95.20, 94.10, 99.12 respectively.

Limitations

  • Benchmark is self-constructed and relatively small (390 problems, 78 knowledge points); no external validation dataset.
  • Code-based rendering only handles geometric/mathematical diagrams; cannot generalize to photorealistic or domain-specific scientific imagery.
  • \(I_\text{max}=5\) is a fixed heuristic; optimal stopping criteria are not learned.
  • Baseline comparison is narrow: only two models natively support text+image generation; LLM baselines need the framework wrapper and cannot be compared head-to-head on image generation alone.
  • No human expert evaluation — all metrics are automated LLM-judge assessments.

Relevance to Harnesses / Meta-Harnesses

MAGMA-Edu is a direct instantiation of a multi-stage agent harness: it wraps arbitrary backbone LLMs with a fixed orchestration shell (GVR loops, structured JSON contracts between agents, threshold-gated termination) that controls agent behavior without modifying model weights. The meta-level parameters \(\Theta\) — prompts, thresholds, evaluation weights — are the harness configuration layer, analogous to how meta-harnesses route tasks and enforce quality gates across sub-agents. The two-stage pipeline design (text harness → image harness, with the output of stage 1 as structured input to stage 2) demonstrates composable harness chaining. The ablation showing that either stage independently provides orthogonal gains validates the modular, independently testable harness architecture pattern.