Multi-agent Self-triage System with Medical Flowcharts

🕒 Published (v1): 2025-11-16 03:48 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TriageMD is a proof-of-concept multi-agent conversational triage system that grounds LLM outputs in 100 clinically validated AMA flowcharts, converting them to graph representations navigated by three specialized agents. The system achieves 95.29% top-3 flowchart retrieval accuracy and 99.10% navigation accuracy across 37,200 synthetic patient interactions, demonstrating that structured clinical protocols can constrain and audit LLM behavior in high-stakes settings.

Problem

General-purpose LLMs used for medical self-triage suffer from hallucination, opacity ("black box" decision paths), and inability to enforce auditable clinical protocols. Existing symptom checkers require health literacy and fail to deliver definitive, case-specific guidance. There is no shared structured representation between LLMs and clinicians that enables both flexibility of free-text interaction and the rigor of standardized triage procedures.

Method

The system converts 100 AMA Family Medical Guide flowcharts into directed graphs stored in a flowchart database. Three LLM-powered agents operate in sequence:

1. Retrieval agent — uses RAG: cosine similarity search over flowchart embeddings retrieves top-\(n\) candidates, then GPT-4o-mini selects the best match via semantic analysis. Combination outperforms either module alone (84.66% vs. 82.14% similarity-only, 78.40% LLM-only at top-1).
2. Decision agent (GPT-4o-mini backbone) — classifies each patient response along four binary axes: On-topic/Off-topic, Answered/Unanswered, Yes/No, Certain/Uncertain. Only "Certain+Correct" responses advance the flowchart; uncertainty holds at the current node.
3. Chat agent — generates empathetic, patient-friendly natural language responses based on the decision agent's output and the current flowchart node.

Evaluation used synthetic datasets generated by four LLMs (GPT-4o, Claude 3 Haiku, Gemini 2.0 Flash-Lite, DeepSeek-Chat) across five patient response styles: brief, descriptive, weak, uncertain, and off-topic.

Key Contributions

• Formalization of medical flowcharts as directed graphs serving as a shared, human-editable knowledge representation between clinicians and LLMs.
• Three-agent pipeline (retrieval → decision → chat) with explicit role separation for triage task decomposition.
• Large-scale synthetic evaluation framework: N=2,000 for retrieval, N=37,200 for navigation, spanning four generator LLMs and five conversational pattern categories.
• Transparency mechanism: each conversation session produces an auditable decision path through the flowchart, enabling clinician review.

Results

• Flowchart retrieval (N=2,000):
• LLM-only: 78.40% ± 6.74% top-1 accuracy
• Similarity search-only: 82.14% ± 7.10% top-1 accuracy
• Combined retrieval agent: 84.66% ± 5.59% top-1 accuracy
• Top-3 similarity results: 95.29% ± 2.88% accuracy
• Top-5 similarity results: 97.71% ± 1.52% accuracy
• Flowchart navigation (N=37,200, overall): 99.10% ± 0.54% acceptable decision accuracy
• Brief responses: 98.11% ± 2.13% acceptable
• Descriptive responses: 99.65% ± 0.14% acceptable
• Weak/ambiguous responses: 99.69% ± 0.22% acceptable
• Uncertain responses: 99.53% ± 0.49% acceptable
• Off-topic responses: 98.26% ± 3.39% correct identification
• Best retrieval agent accuracy by generator: GPT-4o statements → 90.60%; lowest: Gemini Lite → 79.60%

Limitations

• Proof-of-concept only; no evaluation on real patient conversations—all testing used synthetic data generated by LLMs, introducing potential distributional mismatch.
• Flowchart database is fixed at 100 AMA flowcharts; coverage gaps exist for conditions outside the Family Medical Guide.
• Retrieval agent accuracy drops to ~80% top-1 for semantically similar or co-occurring symptoms (e.g., chest pain + dyspnea), a failure mode not resolved by the current architecture.
• No end-to-end multi-turn conversation evaluation; retrieval and navigation are assessed as separate, independent tasks.
• Navigation evaluation does not measure clinical correctness of final triage recommendations—only whether the flowchart is followed correctly.
• Performance varies across synthetic data generators; generalization to real-world linguistic diversity is unvalidated.

Relevance to Harnesses / Meta-Harnesses

TriageMD is a domain-specific harness that wraps LLMs with an external structured controller (the AMA flowchart graph), constraining generative behavior to auditable, predetermined decision paths—an instantiation of the "LLM-as-executor, structured-spec-as-controller" pattern central to harness design. The three-agent decomposition (retrieval → decision → chat) mirrors the modular agent orchestration pattern seen in general-purpose meta-harnesses, but enforces domain-specific invariants (e.g., never advance on uncertain responses) at the harness level rather than relying on the LLM to self-enforce them. The use of synthetic multi-LLM datasets as an evaluation harness for a conversational pipeline—decoupling data generation from the system under test—is directly relevant to anyone designing evaluation infrastructure for multi-agent systems. The transparency mechanism (per-session auditable decision paths) represents a practical design pattern for harnesses operating in regulated or high-stakes domains.