Scaling Equitable Reflection Assessment in Education via Large Language Models and Role-Based Feedback Agents¶

🕒 Published (v1): 2025-11-14 09:46 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

A five-agent LLM pipeline (orchestrated via AutoGen/GPT-4o) automates equitable formative feedback for student reflections, combining rubric scoring, bias detection, metacognitive prompting, and a self-correction step. Evaluated on 336 adult learner reflections across a 12-session AI literacy course, it achieves moderate expert agreement (overall QWK = 0.459, MAE = 0.467) and near-threshold feedback quality (Q(g) = 3.967/5.0) at ~$0.0016 per reflection.

Problem¶

Large and low-resource courses cannot provide timely, individualized formative feedback at scale; instructors lack bandwidth to review every reflection, disproportionately harming lower-proficiency and historically marginalized learners. Existing LLM-based grading systems focus narrowly on score accuracy, ignoring equitable treatment across proficiency bands, metacognitive feedback design, and auditability.

Method¶

A role-based multi-agent pipeline orchestrated with AutoGen runs five specialized GPT-4o agents per reflection: 1. Evaluator: Scores each of four rubric dimensions (0–3) and returns structured JSON with reasoning and improvement suggestions. 2. Equity Monitor: Reviews Evaluator output for biased or exclusionary language and proposes revisions. 3. Metacognitive Agent: Generates 1–2 reflective prompts to encourage self-regulated thinking. 4. Aggregator: Synthesizes outputs into a ≤120-word learner-facing comment, preventing feedback overload. 5. Reflexion Agent: Post-hoc consistency check returning CONFIDENT or REVISE with targeted suggestions.

Evaluator, Equity Monitor, and Metacognitive agents run in parallel; Aggregator and Reflexion run sequentially. Temperature is fixed at 0.3. Fairness is operationalized as ∆MAE — the worst-band error gap between low-ability (score 0–1) and high-ability (score 2–3) learners. No fine-tuning is performed; all inference is zero-shot prompt-based.

Key Contributions¶

End-to-end pipeline integrating rubric scoring, bias-aware feedback generation, and explicit fairness monitoring into a single deployable workflow
Formal fairness objective (∆MAE across proficiency bands) that makes grading disparities measurable and auditable without requiring demographic attributes
Reflexion self-correction layer improving output reliability before learner release
Empirical cost/latency analysis: $0.0016/reflection, 33.35 s end-to-end, ~11× faster than average human grader on scoring alone

Results¶

Scoring accuracy: Overall MAE = 0.467; best on Concept Understanding (0.381), worst on Real-World Application (0.560)
Ordinal concordance: Overall QWK = 0.459 (moderate); strongest on Reflection Questions (µ = 0.483), weakest on Concept Understanding (µ = 0.298)
Inter-rater reliability: Human–Human ICC = 0.573 ± 0.345; Human–AI ICC = 0.410 ± 0.132 (lower but still positive)
Fairness: Aggregate ∆MAE = 0.50; largest gap on Real-World Application (∆MAE = 1.10) disfavoring low-ability learners
Feedback quality: Q(g) = 3.967/5.0; highest on Empathy (4.223), lowest on Actionability (3.760)
Speed: 7.71 s for scoring only vs. ~1.4 min average human; 33.35 s for full feedback pipeline
Cost: $0.0016/reflection; $0.52 for a 28-student, 12-session course

Limitations¶

Evaluated on a single small cohort (28 adult, English-speaking learners, 12 sessions); generalizability to other populations, languages, or subject areas is untested
Low-ability learners still incur substantially higher grading errors (∆MAE = 0.50 aggregate, up to 1.10 on one dimension)
Feedback Actionability and Depth of Insight fall below target (< 4.0/5.0)
Overall Q(g) = 3.967 narrowly misses the stated 4.0 threshold
No causal measurement of learning gains; only proxy measures (grader ratings of feedback quality)
Model drift over time could degrade performance without recalibration
No demographic data collected, so fairness monitoring relies on proficiency-band proxies rather than protected attributes

Relevance to Agentic AI / LLM Agents¶

This paper is a concrete case study of role-specialized multi-agent coordination for a real-world task, demonstrating how decomposing a complex workflow across agents with distinct responsibilities (evaluation, equity checking, metacognitive scaffolding, aggregation, self-correction) improves both output quality and auditability compared to a monolithic LLM call. The Reflexion agent pattern — a post-hoc self-consistency checker that triggers conditional revision — is directly relevant to broader agentic self-improvement loops. The fairness-monitoring architecture illustrates how agent pipelines can embed domain-specific social constraints (equitable treatment) as first-class objectives, a design principle applicable beyond education to any agentic system serving heterogeneous user populations.