Skip to content

An Agentic AI Framework for Training General Practitioner Student Skills

🕒 Published (v1): 2025-12-20 17:26 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A three-agent LLM framework for virtual simulated patients (VSPs) in GP medical education, decomposing the training pipeline into a scenario generator, a persona-driven conversational agent, and a standards-based critic. The framework addresses medical accuracy, persona stability, and rubric-aligned feedback through agentic separation of concerns. A user study (N=14 medical students) confirmed medical realism, effective difficulty calibration, and high feedback utility.

Problem

Existing LLM-based VSPs suffer from hallucinations in patient responses, persona drift beyond 4–6 conversation turns, lack of medically grounded scenario generation at scale, and feedback that is either absent or not aligned with clinical/communication evaluation standards. No prior VSP system integrates all three concerns end-to-end.

Method

A distributed Python backend orchestrates three agents, each with a distinct model assignment:

Generator Agent (GPT-4.1): Multi-step vignette generation — disease selection from a Flemish GP curriculum list, difficulty adjustment via EBM-grounded complicating factors, structured vignette generation from a template, consistency check refinement, and Big Five personality trait translation to textual prompt constraints via a fixed conversion table.

VSP Conversational Agent (LLaMA 4 Scout → LLaMA 4 Maverick → GPT-4o-mini): A 4-step per-turn pipeline: (1) utterance classification (LLaMA 4 Scout) determines response type (vignette-answerable, requires external knowledge, already answered, no question); (2) conditional RAG retrieval via LlamaIndex over an EBM vector database if external medical knowledge is needed; (3) draft response generation (LLaMA 4 Maverick) with vignette + RAG context + personality prompt; (4) constraint post-processing (GPT-4o-mini) enforces personality adherence, removes diagnosis-making, and strips non-verbal artifacts.

Critic Agent (GPT-4o-mini in-session + GPT-4.1 post-session): In-session: concise quick tips on communication. Post-session: MIRS-based communication feedback (25 criteria, 1–5 Likert + quoted evidence, temperature=0.1) and clinical feedback comparing student diagnostic reasoning against EBM gold standard across 7 predefined categories.

Deployment: NiceGUI web dashboard, WebSocket communication, Furhat robot for STT/TTS embodiment.

Key Contributions

  • Configurable, EBM-grounded vignette generation with multi-step consistency verification and difficulty parameterization
  • Multi-step VSP response pipeline with conditional RAG routing and hard-constraint post-processing to reduce hallucinations and persona drift
  • Big Five personality operationalization via prompt-based textual trait descriptions applied at both generation and post-processing stages
  • Dual-mode standards-based automated feedback: MIRS communication scoring with quoted evidence and EBM-aligned clinical reasoning assessment

Results

  • Medical realism: mean 3.93/5 (Case 1), 3.64/5 (Case 2); no inconsistencies or incorrect information reported by any participant
  • Perceived difficulty: 4.29/10 and 6.07/10 vs. targeted mid-level setting, indicating effective calibration
  • Communication feedback composite: 4.23/5 (Case 1), 4.14/5 (Case 2); specific "contains relevant examples" item rated 4.43/5 in both cases
  • Clinical feedback composite: 4.32/5 (Case 1), 4.02/5 (Case 2)
  • Personality consistency rated high; students distinguished between two VSPs (extraversion gap +1.05 points), but 3/5 traits differed from scripted values by ≥0.6 points with statistical significance (|t| ≥ 2.3, p ≤ 0.04)
  • SUS usability: 80.36/100
  • 86% indicated intent to reuse; belief that VSPs prepare for station exams increased from 3.29 → 4.36 pre/post
  • Mean response latency: 3.4s (SD=1.84s, excluding outliers)

Limitations

  • No non-verbal communication capture or generation; limits empathy and communicative realism
  • Personality trait accuracy is limited despite perceived consistency; intended Big Five scores do not reliably translate to perceivable interaction patterns
  • Post-session feedback perceived as too long by 6/14 students
  • In-session quick tips rated comparatively lower and reported as distracting
  • No physical examination simulation capability
  • Small evaluation (N=14, single institution, two fixed disease cases)
  • Generator agent lacks longitudinal case support and does not follow established OSCE case generation best practices systematically

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a multi-agent meta-harness pattern: a central orchestration hub routes tasks across three specialized agents with explicit role boundaries, tiered model selection (accuracy-critical async tasks → GPT-4.1; latency-critical sync tasks → LLaMA 4 + GPT-4o-mini), and conditional tool invocation (RAG triggered only when the utterance classifier determines external knowledge is needed). The per-turn VSP pipeline is itself a micro-harness (classify → retrieve → generate → post-process) nested inside the session-level macro-harness, demonstrating compositional agent orchestration within a harness. The design principle of agentic separation of scenario control, interaction control, and assessment maps directly onto the broader harness literature's concern with modularity and reliability through decomposition.