Skip to content

Code Like Humans: A Multi-Agent Solution for Medical Coding

🕒 Published (v1): 2025-09-04 16:31 UTC · Source: Arxiv · Venue: EMNLP · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLH (Code Like Humans) is a four-agent LLM framework that mirrors the NHS Analyze-Locate-Assign-Verify workflow for ICD-10 medical coding, leveraging the alphabetical index, tabular hierarchy, and coding guidelines as external tools. It is the first system to operate over the full 70K-label ICD-10-CM space without restricting to training-observed codes. CLH matches fine-tuned discriminative classifiers on rare codes while generalizing beyond their closed-set constraint.

Problem

Prior NLP approaches to medical coding (e.g., PLM-ICD) fine-tune encoder models on a subset (~6K) of ICD-10 codes present in MIMIC, producing biased decision boundaries that underpredict rare codes and cannot generalize to the full 70K label space. They also ignore the external resources (alphabetical index, guidelines) that human coders rely on, and struggle with long clinical notes due to token-length limits.

Method

CLH decomposes coding into four sequential agent steps, each instantiated as a separate inference call over the same backbone LLM (thinking-enabled models: DeepSeek-R1 variants, Qwen3-235B, o3-mini, o4-mini):

  1. Evidence extractor — pulls verbatim text snippets from clinical notes that may justify codes.
  2. Index navigator — embeds snippets and retrieves top-10 alphabetical index terms via S-PubMedBert-MS-MARCO; runs in parallel per snippet to yield candidate codes.
  3. Tabular validator — receives {note, chapter guidelines, candidate codes} per ICD chapter in parallel; applies formal coding rules to produce a tentative code set.
  4. Code reconciler — merges parallel outputs, applies instructional notes from the ICD hierarchy, resolves mutually exclusive codes, and produces the final ordered code list.

Candidate codes are retrieved from the alphabetical index at inference time, enabling open-set coverage of all 70K codes without re-training. Guidelines (100+ pages) are chunked per chapter (~3 pages each) and injected only for relevant chapters.

Key Contributions

  • First medical coding system covering the full ICD-10-CM label space (70K codes) without training-time label restriction.
  • Agent pipeline grounded in official NHS coding workflow, with each step corresponding to a defined human coder subtask.
  • Empirical analysis of system "blind spots" across ICD-10 chapters, identifying failure modes (abbreviations, social qualifiers, mental health terms) for future work.
  • Ablation studies quantifying the impact of: candidate set size, guideline enrichment, and thinking-enabled decoding on pipeline component performance.

Results

All results on MDACE (302 inpatient + 52 profee charts, Krippendorff's \(\alpha = 0.97\)):

  • CLH-large (235B, constrained 1K label space): F1 micro = 0.43, F1 macro = 0.28, EMR = 0.02 — comparable to PLM-ICD (F1 micro = 0.48, macro = 0.25) despite no supervised fine-tuning.
  • PLM-ICD advantage is confined to frequent codes due to shared ICU distributional prior; CLH achieves superior F1 macro (rare-code performance).
  • CLH-base (70B, full 70K label space): F1 micro = 0.32, macro = 0.14 — the only system ever evaluated in this realistic open-set setting.
  • Naive LLM baseline (Llama3-70B, constrained): F1 micro = 0.28, macro = 0.18 — substantially below CLH at same scale.
  • Per-agent recall (CLH-large): evidence extractor = 0.62, index navigator = 0.53, tabular validator = 0.47, code reconciler = 0.47; precision rises from 0.06 → 0.40 through the pipeline.
  • Guideline-enriched context improves tabular validator F1 across all candidate set sizes in ablation; adding chapter guidelines over code descriptions yields additional gains.

Limitations

  • Pipeline errors propagate: a missed cue in step 1 cannot be recovered in later steps; no self-refinement loop is implemented yet.
  • Performance degrades as candidate set size grows (confirmed by ablation with hard negatives), limiting scalability of the reconciler in multi-label settings.
  • Evidence extractor fails on abbreviations (e.g., "Hx of CVA"), social/historical qualifiers, medication names, and mental-health terms (F01–F99 chapter recall is low).
  • Thinking-enabled models' reasoning traces are not faithful, limiting interpretability claims.
  • No public dataset covers the full 70K label space; full open-set evaluation remains unverifiable against a gold standard.
  • Inference cost (multi-step reasoning model calls per note) is not benchmarked against PLM-ICD in wall-clock or compute terms.

Relevance to Harnesses / Meta-Harnesses

CLH is a domain-specialized multi-agent harness: it composes four heterogeneous agents (retrieval-augmented, guideline-injected, parallel fan-out, sequential reconciliation) into a fixed orchestration pipeline mirroring a human expert workflow. The architecture pattern — decompose a complex task into typed agent roles, fan out parallel sub-tasks, then merge and reconcile — is a canonical harness pattern applicable beyond medical coding. The ablation methodology (per-agent recall/precision, candidate-set scaling, context-enrichment curves) demonstrates how to instrument and analyze individual harness stages, which is directly transferable to harness evaluation practice. The proposed self-refinement extension (code reconciler output appended as scratchpad, re-invoking all agents) sketches a meta-harness loop.