Skip to content

AdvisingWise: Supporting Academic Advising in Higher Education Settings Through a Human-in-the-Loop Multi-Agent Framework

🕒 Published (v1): 2025-11-07 20:55 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AdvisingWise is a human-in-the-loop multi-agent system for academic advising that automates query processing, information retrieval, and response drafting while requiring advisor review before any response reaches students. Built on LangGraph with a three-phase architecture (query preprocessing → ReAct-style information collection → quality-controlled generation), it achieves 84.2% high-accuracy responses on expert-evaluated queries and improves advisor willingness to adopt AI tooling after direct use.

Problem

High student-to-advisor ratios in higher education create bottlenecks during peak advising periods. Existing LLM-based advising tools either over-automate (cutting out human judgment and trust) or under-deliver on reliability and personalization, failing to handle multi-source inference queries or maintain student-specific context across sessions.

Method

A three-phase multi-agent pipeline implemented in LangGraph, using a tiered model strategy (GPT-4o-mini for lightweight preprocessing agents, GPT-4o for answer generation and quality control):

  1. Query Preprocessing: Three agents—Academic Info Extractor (stores student profile as key-value pairs), Query Rewriter (contextualizes queries using conversation history), and Query Classifier (filters off-topic queries).
  2. Information Collection: A ReAct-style loop between a Thought Agent and an Action Agent, running up to 4 iterations. Actions are: search_knowledge_base (MongoDB vector store with AzureOpenAI embeddings), search_course_db (exact lookup), search_web (Tavily + GPT-4o relevance filter), request_student_info (LangGraph human-in-the-loop pause), or provide_answer. Institutional sources are prioritized for policy queries; student prompting is used for personalization gaps.
  3. Response Generation: Answer Generator produces a five-component structured response (completeness assessment, confidence score, evidence-based answer, source references, identified gaps). Quality Controller validates factual accuracy against retrieved documents, checks citations, and flags hallucinations.

All draft responses—including advisor-only uncertainty notes—route through a Rocket.Chat interface for one-click approval or editing before delivery to students.

Key Contributions

  • Human-in-the-loop system design that keeps both students (context provisioning) and advisors (output validation) in the workflow loop, not as optional overseers.
  • Three-phase multi-agent architecture combining profile-aware query preprocessing, iterative ReAct retrieval, and hallucination-checked response generation.
  • Tiered LLM cost strategy: cheaper model (GPT-4o-mini) for retrieval/classification agents, stronger model (GPT-4o) for final generation.
  • Mixed-methods evaluation combining expert accuracy assessment, LLM-as-a-judge pairwise retrieval comparison, and a qualitative user study with 8 advisors.

Results

  • Expert evaluation on 20 sampled queries (of 100 synthesized): 84.2% (16/19 evaluable) achieved high accuracy; lower-scoring cases reflect conservative refusal-to-answer under uncertainty rather than hallucination.
  • LLM-as-a-judge pairwise comparison: ReAct-style retrieval outperformed standard RAG baseline, with the largest margin on Handbook-Implicit queries (multi-source inference required).
  • User study (\(n=8\) advisors): advisors' initial concerns about reliability and personalization diminished after direct system use; willingness to adopt increased post-interaction.

Limitations

  • Deployed and evaluated only for CS graduate advising at a single private U.S. university; generalizability to other programs/institutions is untested.
  • Expert evaluation covers only 20 of 100 benchmark queries, and 1 query was unevaluable, leaving coverage gaps.
  • ReAct loop is capped at 4 iterations, which may be insufficient for highly ambiguous or multi-step queries.
  • No longitudinal evaluation of advisor trust or student outcomes; user study reflects short-term lab perceptions.
  • Relies on GPT-4o/GPT-4o-mini and external services (Tavily, MongoDB/Azure), introducing cost, latency, and vendor-dependency concerns not analyzed.
  • Advisor review adds latency; the paper does not measure response time relative to unassisted advising.

Relevance to Harnesses / Meta-Harnesses

AdvisingWise is a domain-specific multi-agent harness that pipelines distinct specialist agents—preprocessor, rewriter, classifier, reasoner, retriever, generator, quality controller—under a coordinating orchestration layer (LangGraph) with explicit human checkpoints. The tiered model routing (cheap agents for classification/retrieval, capable models for synthesis/QC) is a canonical harness pattern for managing cost-quality tradeoffs across agent roles. The ReAct-style inner loop with bounded iterations and action-type heuristics illustrates how domain constraints can be baked into a meta-harness's control flow rather than left to unconstrained LLM planning. The advisor-in-the-loop gate demonstrates a mandatory human checkpoint pattern that any safety-critical harness must support.