Skip to content

EduAgentQG: A Multi-Agent Workflow Framework for Personalized Question Generation

🕒 Published (v1): 2025-11-08 12:25 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EduAgentQG is a five-agent collaborative framework for generating personalized mathematics questions aligned with structured educational goals. It uses an iterative plan→generate→evaluate→refine loop with explicit diversity controls—multiple question directions from a Planner and multi-perspective rewrites by a Writer—rather than relying on inherent model randomness. On two math datasets it outperforms single-agent and multi-agent baselines in diversity, goal consistency, and overall quality.

Problem

Existing question generation methods—template-based, single-agent CoT, and prior multi-agent pipelines—produce questions with unstable quality, limited diversity, and weak alignment with structured educational objectives (grade, knowledge concept, difficulty, core competency). Current multi-agent approaches rely on model randomness for diversity and use aggregated or implicit evaluation scores, which obscure specific dimensional deficiencies and prevent fine-grained quality control.

Method

EduAgentQG orchestrates five specialized LLM agents through a closed iterative loop:

  1. Planner decomposes educational goal \(E = \{G, K, D, S\}\) into a structured plan \(P(E, \text{RAG}) = (\text{Plan}, \text{Dir})\) with three explicit question directions \(\text{Dir} = \{\text{dir}_1, \text{dir}_2, \text{dir}_3\}\) retrieved via RAG over a knowledge graph and curriculum standards.
  2. Writer generates one candidate per direction \(Q^{(0)} = \{q_1^{(0)}, \ldots, q_N^{(0)}\}\), then iteratively rewrites failing candidates conditioned on evaluator feedback: \(q_i^{(t+1)} = \text{Writer}(q_i^{(t)}, \mathcal{F}(q_i^{(t)}))\) for up to \(T_{\text{rewrite}}\) rounds.
  3. Solver applies binary scoring over logical constraints \(C_{\text{logic}}\), emitting pass signal \(p_i^S\) and ranking score \(r_i^S\).
  4. Educator applies binary scoring over educational constraints \(C_{\text{edu}}(D, G, S)\) using RAG-retrieved reference questions as difficulty anchors, emitting \(p_i^E\) and \(r_i^E\).
  5. Checker performs final verification (answer correctness, clarity) on \(Q_{\text{pass}} = \{q_i \mid p_i^S = 1 \wedge p_i^E = 1\}\), selecting \(q^* = \text{Checker}(\arg\max_{q_i \in Q_{\text{pass}}} r_i^S + r_i^E)\).

A question is accepted only when it passes all binary checks across both Solver and Educator dimensions simultaneously; partial passage does not suffice.

Key Contributions

  • Five-role multi-agent workflow (Planner, Writer, Solver, Educator, Checker) covering the full plan→generate→evaluate→refine lifecycle for educational question generation.
  • Explicit diversity mechanism: Planner produces three structurally distinct directions; Writer performs multi-perspective iterative rewrites from distinct reasoning paths and contexts.
  • Per-dimension binary scoring (not aggregated scores) with dual ranking (\(r^S + r^E\)), enabling strict and interpretable quality control.
  • Dual-level quality control: strict pass/fail gating at evaluation stage combined with ranking-based selection and iterative Writer refinement.
  • Evaluated on a custom K-9 mathematics dataset (10,273 questions, 634 knowledge concepts, 16 competencies) split into MathChoice (489 goals) and MathBlank (500 goals).

Results

  • EduAgentQG achieves the highest win rate of 0.53 in human evaluation (consistency score 8.42), versus COT (0.39/8.09), COT-N (0.33/8.18), REACT (0.41/8.24), EQPR (0.47/7.97).
  • Win rate pairwise matrix shows EduAgentQG wins against EQPR at 0.63, REACT at 0.82, COT-N at 0.84, and COT at 0.74.
  • Outperforms all baselines on three axes: question diversity, educational goal consistency, and overall quality across both MathChoice and MathBlank subsets.

Limitations

  • Evaluation is restricted to mathematics (K-9); generalizability to other STEM or humanities domains is untested.
  • Human evaluation uses win-rate comparisons; no automated diversity metric (e.g., semantic distance, structural variety) is reported with precise numbers in the provided text.
  • \(T_{\text{rewrite}}\) (maximum rewrite iterations) is not ablated; cost and latency of the iterative loop are not analyzed.
  • The RAG knowledge base and question bank must be pre-populated with curriculum-aligned content, limiting deployment to settings with such resources.
  • Binary pass/fail across all dimensions simultaneously may be overly strict, potentially discarding near-valid candidates without a softer fallback.

Relevance to Harnesses / Meta-Harnesses

EduAgentQG is a concrete domain-specific multi-agent harness: it encodes a fixed workflow topology (plan→write→eval→refine) with explicit agent roles, inter-agent communication contracts (binary signals + ranking scores + natural-language feedback), and a loop-until-pass termination condition. The pattern—a Planner decomposing goals, a Generator producing candidates, Critic agents scoring across dimensions, and a Checker gating output—is a reusable meta-harness architecture directly analogous to general-purpose agentic pipelines. The use of RAG at two stages (Planner and Educator) and the strict binary gate with ranking fallback are design patterns transferable to any harness that must enforce multi-dimensional quality constraints over LLM-generated artifacts.