Skip to content

Conversational Education at Scale: A Multi-LLM Agent Workflow for Procedural Learning and Pedagogic Quality Assessment

🕒 Published (v1): 2025-07-07 22:56 UTC · Source: Arxiv · Venue: EMNLP 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WikiHowAgent is a multi-LLM agent workflow that orchestrates teacher, learner, interaction manager, and evaluator agents to generate and assess procedural learning conversations at scale. The system produces 114,296 teacher-learner dialogues grounded in 14,287 WikiHow tutorials across 17 domains, benchmarked across 8 LLMs. Evaluation reveals persistent gaps between LLM-based rubric scoring and human judgment, especially on depth, engagement, and naturalness.

Problem

Existing LLM-based educational systems lack diversity and scale in course content, rely on single-agent setups, and provide no comprehensive framework for assessing pedagogic quality. This limits insights into how LLMs perform as teachers or learners across varied instructional domains.

Method

The workflow is structured as a conversational graph where each node is a dialogue turn and edges encode agent transitions. Four components operate in sequence per tutorial \(T = [t_1, t_2, \ldots, t_n]\):

  1. Teacher Agent — provides step-by-step instructions and answers clarification questions.
  2. Learner Agent — simulates comprehension, generates follow-up questions or acknowledgments.
  3. Interaction Manager — tracks state \(S\) (current step, clarification flag, completion), routes control to TEACHER / LEARNER / END nodes.
  4. Evaluator Agent — scores the finished conversation via computational metrics (BLEU, METEOR, ROUGE, BERTScore, diversity, question rate, completion) and 8 rubric-based metrics (clarity, truthfulness, engagement, coherence, depth, relevance, progress, naturalness) using an LLM-as-judge approach.

Two evaluation regimes are defined: homogeneous (same LLM for all three agents) and heterogeneous (fixed teacher/evaluator LLM, varied learner LLM). Human alignment is measured with Pearson's \(r\), Spearman's \(\rho\), and Kendall's \(\tau\) on a 175-conversation sample rated by two annotators.

Key Contributions

  • WikiHowAgent dataset: 114,296 teacher-learner conversations from 14,287 WikiHow tutorials, spanning 17 domains and 727 topics in a hierarchical knowledge graph.
  • Multi-LLM agent workflow with an explicit interaction manager and state-tracked conversational graph, benchmarked on 8 LLMs (DeepSeek-7B, Qwen2-7B, Gemma-7B, OLMo2-7B, OpenChat-7B, Llama3-8B, Phi4-14B, GPT-4).
  • Comprehensive evaluation protocol combining 7 computational metrics, 8 rubric dimensions, and human-LLM alignment correlation analysis.
  • Full open-source release of datasets and implementation.

Results

  • Homogeneous setting: mean completion \(\mu = 95.16\%\), diversity \(\mu = 80.14\%\), BERTScore \(\mu = 70.42\%\); high inter-model variance in question generation (\(\delta = 20.70\%\)).
  • Heterogeneous setting: completion improves to \(\mu = 98.97\%\); BLEU \(+3.23\%\), ROUGE \(+1.50\%\), diversity \(+0.86\%\) vs. homogeneous; question rate drops \(-8.89\%\); standard deviations contract across most metrics.
  • Rubric scores: Relevance (\(\mu = 4.70\)) and Clarity (\(\mu = 4.61\)) are highest; Truthfulness, Naturalness, and Depth show the most variability (\(\delta = 0.7, 0.5, 0.47\)).
  • Human alignment: Clarity, Progress, Coherence, Relevance correlate significantly (\(p < 0.05\)) with human judgments; Depth, Engagement, Naturalness, and Truthfulness show low/non-significant correlation.
  • LLM evaluators cluster ratings tightly around high medians; human raters produce lower, wider distributions — indicating systematic LLM positivity bias.

Limitations

  • Human evaluation conducted on only 25 tutorials (175 conversations), limiting statistical power.
  • All tutorials sourced from a single platform (WikiHow); domain coverage may not generalize to formal curricula or technical STEM content.
  • Learner agent simulates engagement via question generation only; actual learner modeling (knowledge state, misconceptions) is absent.
  • Heterogeneous experiments fix teacher/evaluator to OpenChat; other teacher-model pairings are unexplored.
  • LLM-as-judge positivity bias is identified but not corrected.
  • No real human learners in the loop; system validity for actual learning outcomes is untested.

Relevance to Harnesses / Meta-Harnesses

WikiHowAgent is a concrete domain-specific multi-agent harness: it chains four specialized LLM agents (teacher, learner, interaction manager, evaluator) under a controlling workflow that manages state, branching, and termination — the architectural pattern central to harness design. The interaction manager role is notable as an explicit orchestrator that routes between agents based on runtime state, analogous to the dispatch logic in a meta-harness. The two-level evaluation framework (automated pipeline metrics + rubric-based LLM judge) mirrors harness-level quality assurance patterns, and the homogeneous vs. heterogeneous benchmarking methodology offers a reusable template for evaluating how model substitution affects multi-agent pipeline outputs.