Skip to content

ClinNoteAgents: An LLM Multi-Agent System for Predicting and Interpreting Heart Failure 30-Day Readmission from Clinical Notes

🕒 Published (v1): 2025-12-08 01:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ClinNoteAgents is a three-agent LLM pipeline (Qwen3-14B) that transforms unstructured hospital discharge notes into structured risk-factor representations and clinician-style summaries for predicting 30-day heart failure readmission. It requires no labeled training data or predefined ontologies, making it viable in data-limited settings. On 3,544 MIMIC-III notes, it achieves ≥90% conditional extraction accuracy for most cardiorespiratory vitals and preserves most predictive signal with 60–90% text compression.

Problem

~80% of EHR information lives in free-text clinical notes, yet HF readmission models predominantly rely on structured fields (labs, ICD codes, demographics). Existing NLP approaches require expert-crafted ontologies, manual annotation, or predefined SDOH taxonomies that are coarse, annotation-heavy, and unavailable in resource-limited healthcare settings. No unified framework jointly extracts both clinical measurements and social determinants of health (SDOHs) from discharge notes for simultaneous risk-factor mining and readmission prediction.

Method

Three specialized LLM agents, all Qwen3-14B with chain-of-thought ("thinking mode") enabled, operate in sequence:

  1. Risk Factor Extractor — zero-shot structured extraction from discharge notes using domain-specific prompts; outputs JSON covering charted SDOHs (gender, age, language, marital status), uncharted SDOHs (alcohol, tobacco, drug use, housing, employment, social support, transportation, parental status), vitals, chief complaint, and diagnoses.

  2. Risk Factor Normalizer / Labeler — two-stage normalization: a first LLM generates cohort-specific category labels via k-medoids clustering (k=200) of raw free-text entries; a second LLM assigns each extracted value to one of these categories. Evaluated against MIMIC-SBDH and LLM-SDOH taxonomies.

  3. Note Summarizer — produces either (a) an overall summary retaining numerical values or (b) a no-number summary replacing all numbers with qualitative descriptors, both targeting readmission-relevant clinical signals.

Downstream evaluation uses three classifiers on the summaries: TF-IDF + logistic regression, ClinicalBERT, and LoRA-finetuned Qwen3-8B. Statistical association analysis uses logistic regression (continuous variables) and chi-square tests (categorical) against binary 30-day readmission labels.

Key Contributions

  • Fully zero-shot, modular multi-agent pipeline for joint clinical and SDOH extraction from discharge notes — no task-specific training or annotation required.
  • LLM-driven normalization that generates cohort-specific SDOH categories (instead of mapping to fixed external taxonomies), enabling correlation analysis on otherwise unanalyzable free-text.
  • Empirical demonstration that 60–90% text compression via LLM summarization retains most predictive signal (AUROC drops of 0.01–0.10 across three classifiers).
  • LLM-as-a-judge evaluation framework applied to diagnoses, with conditional accuracy of 62.67% against ICD-9 codes.
  • Statistical identification of age, weight, and blood pressure as significant readmission predictors, and housing as the only significant SDOH, from LLM-extracted structured data.

Results

  • Vital extraction (conditional accuracy within tolerance): SpO2 94.16%, RR 93.77%, BP SYS 90.85%, BP DIA 91.24%, HR 88.59%, Temperature 84.24%; Height (68.50%) and Weight (57.90%) substantially lower due to unit heterogeneity.
  • Charted SDOH extraction accuracy: Gender 99.94%, Age 93.38%, Language 88.89%, Marital Status 77.89%.
  • Diagnosis extraction (LLM-as-judge): mean similarity score 3.04/5, conditional accuracy 62.67%, absolute accuracy 25.25% (model extracts ~5.9 diagnoses vs. 15.1 ICD-9 codes per patient).
  • Readmission prediction AUROC (raw notes baseline): LR 0.6535, ClinicalBERT 0.6095, LoRA-Qwen3-8B 0.6064.
  • No-number summary (61% compression): LR 0.6434, ClinicalBERT 0.6046, LoRA 0.5634 — closest to raw baseline.
  • Overall summary (83% compression): LR 0.5866, ClinicalBERT 0.5986, LoRA 0.5588.
  • Structural extraction summary (91% compression): LR 0.5735, ClinicalBERT 0.5708, LoRA 0.5595 — largest drop.
  • Housing was the only statistically significant SDOH (chi-square p=0.012); age (p=0.008), weight (p=0.010), and BP (p<0.001/0.037) were significant clinical predictors.

Limitations

  • Extraction fidelity evaluated against structured EHR as surrogate ground truth, not clinician-verified annotations.
  • No formal clinician evaluation of extraction or summarization quality.
  • Summarization did not improve predictive performance over raw notes — only preserved it.
  • Low extraction coverage for some variables (Height: 4.03%, Language: 6.07%) due to under-documentation, not extraction failure; association analyses on these variables are subject to selection bias.
  • SDOH under-documentation in discharge notes limits reliable detection of many social factors (most non-housing SDOHs not significant despite known epidemiological importance).
  • Single institution (MIMIC-III), single discharge note per admission; no temporal or multi-note modeling.
  • LLMs prone to hallucination and unit-conversion errors (particularly for anthropometric measurements).

Relevance to Harnesses / Meta-Harnesses

ClinNoteAgents is a concrete instantiation of a domain-specific multi-agent harness: a fixed pipeline of specialized LLM agents (extractor → normalizer/labeler → summarizer) each with distinct prompts, roles, and evaluation criteria, orchestrated without a meta-controller but with explicit handoffs between stages. The two-stage normalization sub-pipeline (LLM generates categories, second LLM assigns labels) is itself a micro-harness pattern — using one model's output as the schema for another's classification. The paper demonstrates the harness design principle of decomposing a complex NLP task into subtasks matched to LLM generative strengths (zero-shot extraction, open-vocabulary normalization, abstractive summarization), with each agent evaluated independently, which is directly analogous to capability-segregated meta-harness architectures in the research literature.