Agentic AI-based Framework for Mitigating Premature Diagnostic Handoff and Silent Hallucination in Healthcare Applications¶

🕒 Published (v1): 2026-06-16 15:39 UTC · Source: Arxiv · link

Why this paper was selected

Buyya co-author; agentic healthcare harness addressing hallucination and diagnostic handoff

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes a neuro-symbolic multi-agent clinical triage framework that replaces probabilistic LLM routing with deterministic orchestration gates to prevent two failure modes: premature diagnostic handoff and silent hallucinations. Two symbolic gates—one enforcing complete OLDCARTS symptom collection and one screening diagnostic outputs via semantic entropy—are integrated into an AutoGen-based pipeline. On 150 MedQA-derived test cases, the full architecture achieves 49.3% diagnostic precision, +11.3 pp over the unconstrained baseline.

Problem¶

Unconstrained conversational LLM agents in clinical settings fail in two systematic ways: (1) they proceed to diagnosis before collecting a complete symptom history (premature handoff), and (2) they emit confident but divergent or fabricated diagnoses without any uncertainty signal (silent hallucination). Existing mitigations—RAG, chain-of-thought, LLM-as-a-judge—are probabilistic and cannot guarantee intake completeness or flag divergent outputs deterministically.

Method¶

A three-phase pipeline is built on the AutoGen multi-agent framework using Llama 3.3 70B via Groq:

Phase 1 — M1 (Neuro-Symbolic OLDCARTS Gate): A TriageNurse agent elicits symptoms turn-by-turn; each question is tagged [FIELD:X] for one of the 8 OLDCARTS dimensions. A regex-based parser updates a Boolean state vector $V = \{V_{\text{Onset}}, \ldots, V_{\text{Severity}}\}$. The completeness score $\sigma = \sum_{i=1}^{8} V_i$ must equal 8 before the pipeline transitions to diagnosis; otherwise the conversation is routed back.

Phase 2 — M2 (Semantic Entropy UQ Gate): $K=5$ Senior Diagnostician instances are sampled in parallel at temperature $T=0.7$. Their diagnosis labels are clustered by bidirectional NLI entailment ($\tau_{\text{NLI}}=0.7$): labels $a,b$ share a cluster iff $P(\text{ENT}|a\!\to\!b) \geq \tau_{\text{NLI}}$ and $P(\text{ENT}|b\!\to\!a) \geq \tau_{\text{NLI}}$. Normalized Shannon entropy is then: $$H = -\frac{1}{\log_2|C|}\sum_{j=1}^{|C|} p_j \log_2 p_j, \quad H \in [0,1]$$ Cases with $H > \tau = 0.5$ are flagged to the Medical Safety Supervisor.

Phase 3 — Deterministic Diagnosis + Recursive Safety Review: A separate diagnostician at $T=0.0$ generates a final report (with optional check_drug_interaction() tool call). The Medical Safety Supervisor reviews it alongside the entropy signal; on rejection it appends structured feedback to the conversation history for up to $R_{\text{max}}=3$ revision attempts.

Patient agents for evaluation are simulated using llama-3.1-70b-instruct on MedQA-derived cases.

Key Contributions¶

Deterministic neuro-symbolic OLDCARTS gate (M1) enforcing $\sigma=8$ before diagnostic handoff, implemented via regex tag parsing on agent output.
Semantic entropy UQ gate (M2) adapting the Kuhn et al. SE estimator to multi-label differential diagnosis via NLI clustering, integrated as a hard routing criterion.
Recursive safety supervision loop with up to 3 revision attempts before approving patient-facing output.
Empirical demonstration that $\sigma$ and $H$ are negatively correlated ($r=-0.181$, $p<0.05$), suggesting structured intake causally reduces diagnostic uncertainty.

Results¶

Full Architecture (FA): 49.3% diagnostic precision on $N=150$ cases.
Baseline (B, no gate, $K=1$): 38.0% — FA is +11.3 pp absolute.
Ablation A ($K=1$, M1 only): intermediate; M1 alone accounts for a portion of the gain; combined M1+M2 (FA) is best.
Statistically significant negative correlation between OLDCARTS completeness $\sigma$ and semantic entropy $H$: $r=-0.181$, $p<0.05$.
Results reported across $N \in \{50, 100, 150\}$ test cases; $N=150$ is the primary setting.

Limitations¶

Diagnostic precision of 49.3% remains low in absolute terms; the system is not clinically deployable as evaluated.
Evaluation uses LLM-based patient simulators and a LLM ClinicalJudge, introducing circularity in the correctness label.
The check_drug_interaction() tool is a "static prototype" — not connected to a real pharmacological database.
The OLDCARTS gate requires structured tag parsing from TriageNurse output; robustness to tag omission or malformation is not analyzed.
Tested only on MedQA-derived cases; no real patient data or prospective clinical evaluation.
The semantic entropy threshold $\tau=0.5$ and NLI threshold $\tau_{\text{NLI}}=0.7$ are fixed; no sensitivity analysis is provided.
Evaluation metric is binary top-$k$ match, not clinically validated diagnostic accuracy.

Relevance to Harnesses / Meta-Harnesses¶

This paper is a direct instantiation of a deterministic multi-agent harness: the orchestration layer is not another LLM but a symbolic program that gates, routes, loops, and retries agent calls based on verifiable conditions ($\sigma=8$, $H\leq\tau$, revision count). The pattern of replacing probabilistic LLM-as-a-judge routing with hard symbolic guards is a key design principle for robust harnesses, and the recursive review loop (max retries, structured rejection feedback propagated back into context) is a reusable harness primitive. The bidirectional NLI clustering step before entropy calculation also demonstrates how harnesses can canonicalize LLM outputs into verifiable structured representations before making control-flow decisions. This work offers a concrete, domain-specific existence proof that neuro-symbolic gates at pipeline transition points improve end-to-end pipeline reliability.