Multi-Agent Collaborative Framework For Math Problem Generation

🕒 Published (v1): 2025-11-06 01:24 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A multi-agent collaborative framework is proposed for automatic math question generation (AQG) in intelligent tutoring systems, pairing iterative critique (Teacher-Critic Cycle) and collective consensus (Collective Consensus) workflows with a Bloom's Taxonomy-guided self-curation step. Curated agentic methods outperform single-agent baselines on difficulty matching and pedagogical quality, though gains are incremental. Non-curated agentic generation alone underperforms baselines, highlighting that structured selection is necessary.

Problem

Existing AQG systems struggle to precisely control question complexity and cognitive demand: single-model approaches (zero-shot or few-shot) cannot reliably calibrate difficulty across easy/medium/hard tiers, and harder questions consistently degrade in quality. There is also no established automated evaluation pipeline for multi-dimensional question quality in math education.

Method

Two multi-agent workflows are built on top of role-specialized LLM agents:

Teacher-Critic Cycle (TCC): A Teacher agent generates a question-answer pair; a Generic Critic iteratively provides feedback on clarity, relevance, and difficulty alignment over 2–5 rounds. Variants test AutoCoT vs. explicit Solution Generation prompting.

Collective Consensus (CC): 2–4 Versatile Agents sequentially contribute (generate, revise, or endorse) with randomized decoding parameters (temperature/seed) to encourage diversity; a Consensus CEO agent selects the final pair after 2–5 rounds.

Self-Curation: A Bloom Agent scores each candidate question 1–5 on cognitive demand aligned to Bloom's Taxonomy tiers (lower/middle/upper). Questions failing the target tier are discarded; retained questions must exhibit strong Bloom alignment. This is compared against Random Curation (RC).

Inputs: KC name (Common Core), example questions, target difficulty. Evaluated on five LLM-judged criteria (Relevance, Importance, Clarity, Difficulty Matching, Answerability) scored 1–5 using a G-Eval-style framework via GPT-4.

Key Contributions

• Two novel multi-agent AQG frameworks (TCC and CC) designed specifically for ITS-contextualized math problem generation.
• Bloom's Taxonomy-guided self-curation method that prioritizes cognitive-demand alignment over syntactic similarity.
• An automated 5-dimensional evaluation framework (Relevance, Importance, Clarity, Difficulty Matching, Answerability) for math AQG.
• Empirical analysis of inference-time computation scaling: number of rounds and agents exhibit diminishing returns, not monotonic improvement.

Results

• Best curated TCC: Clarity 3.75, Relevance 4.92, Importance 4.70, Difficulty Matching 4.88, Answerability 4.94, Avg. 4.64.
• Best curated CC: Clarity 3.60, Relevance 4.99, Importance 4.76, Difficulty Matching 4.96, Answerability 4.94, Avg. 4.65.
• Baseline Zero-Shot: Avg. 4.40; Baseline Few-Shot: Avg. 4.42.
• Non-curated CC (RC) scores Avg. 4.34 — below both baselines, demonstrating that multi-agent generation without curation is harmful.
• Difficulty Matching degrades monotonically as target difficulty increases across all methods; hard questions are the hardest tier to calibrate.
• Baseline Zero-Shot outperforms Baseline Few-Shot and several agentic variants on Difficulty Matching, suggesting few-shot examples introduce difficulty-alignment bias.
• Prompting strategy (empirical vs. empirical-matched vs. simple) has negligible effect on outcomes (Difficulty Matching range: 4.61–4.71 for TCC/CC).
• Ceiling effect observed in LLM-based evaluation scores (GPT-4 evaluator), limiting discriminative power.

Limitations

• Evaluation relies entirely on GPT-4-based automated scoring; known ceiling effects and potential LLM-evaluator biases are unaddressed (human evaluation is future work).
• Improvements over baselines are incremental, not substantial.
• Scaling inference compute (more rounds/agents) does not reliably improve quality and can degrade it; optimal hyperparameters not found.
• Few-shot prompting strategies yield negligible benefit — the mechanism by which in-context examples influence difficulty alignment is not well understood.
• Dataset is limited to middle-school math (ASSISTments/Problem Bodies); generalizability is untested.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of an agentic meta-harness pattern: specialized role agents (Teacher, Critic, Versatile, CEO) are composed into iterative pipelines (TCC, CC) whose outputs are then filtered by a meta-level quality agent (Bloom Agent) — a judge-and-curate layer structurally analogous to verifier/selector stages in broader meta-harness architectures. The finding that non-curated agentic generation underperforms baselines, while Bloom-curated generation surpasses them, is strong empirical evidence that the curation/selection stage — not the generation stage — is the critical value-add in multi-agent harnesses. The diminishing-returns result on round/agent scaling is directly relevant to harness designers deciding compute allocation across pipeline stages.