Teaching LLMs to Recommend and Defer in Underrepresented Epilepsy Care

🕒 Published (v1): 2026-06-30 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MANANA is a non-parametric, prompt-learning multi-agent framework that adapts an LLM to local anti-seizure medication prescribing norms in Uganda without weight updates. It learns auditable prescribing rules from a small training set by routing observed errors through a Predictor–Inspector–Architect loop, and adds Bayesian prompt averaging (BPA) over the learned prompt trajectory to produce calibrated deferral signals. On two independent Ugandan pediatric epilepsy cohorts, MANANA outperforms prompt-optimization baselines and enables 95–99% top-1 precision selective prediction.

Problem

Medical LLM benchmarks are dominated by high-income clinical settings (e.g., MIMIC-IV), leaving prescribing norms, drug availability, and follow-up patterns in low-resource contexts unrepresented. Direct LLM prompting in Uganda fails not because of missing medical knowledge but because the model applies Western prescribing priors. Existing prompt-optimization methods (TextGrad, DSPy, ExpeL) produce global rewrites without per-rule case provenance, and none provide a calibrated uncertainty/deferral signal essential when specialist attention is scarce.

Method

MANANA adapts an LLM to local practice via a three-agent loop operating over a small patient-level training set (~50 patients, 150 visits):

1. Predictor: given patient history \(x_i\) and current memory state \(m_{t-1}\), returns three candidate regimens \(\hat{y}_{i,1:3}\).
2. Inspector: compares candidates against physician prescription \(y_i\), producing a structured report \(\rho_i\) and candidate learning \(c_i\).
3. Architect: aggregates an append-only evidence buffer \(C_t\) and batch reports \(R_t\) to update memory \(m_t\). A candidate rule is committed only when it recurs across \(N \geq 2\) distinct patients.

Two memory variants: - MANANA-Single: memory is a flat list of evidence-gated prescribing rules; at most one rule appended per round. - MANANA-Multi: memory is a set of specialist agents (max \(A_{\max}=5\)); the Architect can SPAWN, EDIT, or PRUNE agents, each responsible for extracting a specific clinical signal.

Bayesian Prompt Averaging (BPA) treats the learned prompt trajectory \(m_1,\ldots,m_T\) as a Bayesian model ensemble. Posterior weights are computed on a held-out validation set via temperature-smoothed log-likelihood: $\(w_k = \frac{\exp\{\ell_k/\tau\}}{\sum_q \exp\{\ell_q/\tau\}}, \quad \ell_k = \log p(D_\text{val} \mid m_k)\)$ Since the LLM returns ranked candidates rather than probabilities, a candidate-position prior \(\pi = (0.85, 0.11, 0.04)\) is estimated from training data. The posterior predictive is: $\(p_\text{BPA}(y \mid x) = \sum_{k=1}^{K} w_k \sum_{j=1}^{3} \pi_j \mathbf{1}[\hat{y}_{k,j} = y]\)$ The resulting confidence scores enable selective prediction: cases below a confidence threshold are deferred to specialist review.

Key Contributions

• Non-parametric multi-agent prompt-learning framework (MANANA) that adapts LLM prescribing behavior to low-resource clinical settings without weight updates.
• Evidence-gated memory architecture: candidate learnings must recur across distinct patients before being committed, distinguishing MANANA from prompt-rewrite methods (TextGrad) that update global rules without case provenance.
• Bayesian prompt averaging (BPA): converts a learned prompt trajectory into prescription likelihoods and a deferral signal without requiring the LLM to output calibrated probabilities.
• Evaluation on two independently collected Ugandan pediatric epilepsy cohorts (Cohort A: 332 patients/1,040 visits; Cohort B: 367 patients/1,509 visits), with Cohort B as a zero-shot transfer test.
• CONSILIUM: an expert-designed multi-agent comparator built from neurologist audit of single-agent failures, used as a strong human-engineered reference.

Results

• EM@3 (Top-3 exact match) — MANANA-Multi on Cohort B monotherapy: 87.3% (V1), 95.1% (V2), 93.2% (V3) vs. Base Prompt 53.1%/57.6%/58.4% and Single-agent 76.7%/82.6%/83.8%.
• Polytherapy Cohort B: MANANA-Multi 40.2%/61.1%/69.8% vs. Base Prompt 24.8%/29.1%/30.4% and Single-agent 39.4%/49.4%/54.0%.
• BPA vs. majority vote (Cohort B): Top-1 improves from 73% to 75%; confidence gap between correct and incorrect top-1 predictions improves from 0.173 to 0.214.
• Selective prediction (Cohort B): top 25% most-confident cases → 99% top-1 precision; top 50% → 95% top-1 precision; BPA improves Top-3 accuracy by 4–8 pp over prompt-optimization baselines.
• Transfer: trained solely on Cohort A (70-patient pool), MANANA generalizes to Cohort B without additional adaptation.

Limitations

• Evaluation is limited to a 10-drug action space in two Ugandan referral centers; generalization to other LMICs or different formularies is untested.
• The candidate-position prior \(\pi = (0.85, 0.11, 0.04)\) is estimated from a small training subset; BPA calibration inherits this approximation.
• Performance on polytherapy remains substantially lower than monotherapy (exact-match for full drug combinations is harder), especially at Visit 1.
• Learning set is small (50 patients); behavior under smaller or noisier sets is not fully characterized in the main text.
• Experiments use gpt-oss-120b; smaller-model (20B) transfer is examined in an appendix but not the main results.
• Ground truth is physician-prescription agreement, not patient outcomes; high EM@3 on unchanged-regimen visits could inflate scores (checked via a copy-baseline ablation, but not fully resolved in main text).

Relevance to Agentic AI / LLM Agents

MANANA is a concrete instantiation of a multi-agent learning loop where agents (Predictor, Inspector, Architect) collaboratively update a shared, inspectable memory state — directly relevant to work on LLM agent self-improvement and tool-augmented memory. The evidence-gated commit mechanism and the separation between error diagnosis and memory update are novel regularization strategies that contrast with standard prompt-optimization pipelines and speak to open questions about how agents should learn from feedback without overfitting to noise. BPA as uncertainty-aware ensemble over learned prompt trajectories offers a general recipe for adding calibrated deferral to any prompt-learning agent, applicable beyond clinical settings. The work grounds agentic LLM adaptation in a real deployment constraint — no weight updates, small local data, scarce specialist oversight — which makes it a useful case study for designing trustworthy autonomous agents in resource-constrained environments.