HARMON-E: Hierarchical Agentic Reasoning for Multimodal Oncology Notes to Extract Structured Data

🕒 Published (v1): 2025-12-22 20:38 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HARMON-E is a hierarchical agentic LLM framework for end-to-end structured data extraction from real-world oncology EHR notes at patient scale. It decomposes extraction into modular retrieval, LLM synthesis, and deterministic collation stages, achieving an average F1 of 0.93 across 103 clinical variables over 400,000+ documents from 2,250 patients. Integration into a production curation platform yields a 94.1% direct manual approval rate.

Problem

Prior oncology NLP systems extract narrow concept sets (staging, a single biomarker class, one treatment line) from single documents using single-pass encoders or rule-based pipelines. They cannot perform patient-level synthesis across hundreds of heterogeneous, longitudinal, often contradictory documents; they lack cross-entity dependency resolution, large-scale deduplication, and curator-ready validation for a comprehensive clinical data dictionary (103 attributes, 16 entity types).

Method

HARMON-E constructs a three-stage, configurable pipeline per entity type:

1. Retrievers — sentence-bounded chunking (max M chars, one-sentence overlap) fed into either a Vector Retriever (embedding cosine-similarity top-k) or a Regex Retriever (pattern matching). Retrieval size k is tuned per attribute (e.g., k=16 for sparse surgery mentions, k=4 for localized biomarker result lines).

2. LLM Synthesizer — GPT-4 is prompted with entity-specific schema instructions to emit typed JSON attribute-value pairs from retrieved chunks.

3. Collator — deterministic post-processing that canonicalizes values, enforces typed schemas (categorical, date, numeric), deduplicates across documents, resolves conflicts via domain precedence rules, and manages cross-entity dependencies (e.g., medication collation waits for finalized diagnosis date).

Four canonical pipelines are selected by entity complexity: Single-Step (one-pass, localized entities like Biomarker), Multi-Step (enumerate subtypes then detail, e.g., Medication), Topical Extraction (partition documents by report type, apply specialized prompts), and Sequential Document Analysis (per-document extraction preserving chronological provenance, e.g., Staging). Scanned PDFs are normalized to Markdown via a vision-language model before entering the pipeline. Evaluation uses root-based alignment (exact match on a key anchor attribute) or weighted alignment (weighted attribute-score threshold τ) depending on entity type, with expert manual review of the 50 highest-disagreement cases per entity as a secondary protocol.

Key Contributions

• First end-to-end agentic LLM framework for comprehensive oncology data extraction (16 entity types, 103 attributes) validated at production scale (400,000 notes, 2,250 patients).
• Modular Retriever → Synthesizer → Collator architecture with four interchangeable pipeline strategies selectable by entity complexity.
• Explicit cross-entity dependency management (ordered collation) enabling inference of implicit variables (e.g., treatment end date derived from adverse event narrative).
• Novel evaluation methodology combining entity-level recall/precision, attribute-level accuracy, weighted/root-based alignment, and clinician acceptance scoring rather than standard NER metrics alone.
• Demonstrated production integration: 94.1% direct abstractor acceptance rate, substantially reducing manual curation cost.

Results

• Overall: Average F1 = 0.93 across 103 variables; 100/103 variables exceed F1 = 0.85.
• Top entities (F1): Nicotine Use Status = 1.00, Patient Status = 1.00, Clinical Trial = 0.98, Biomarker = 0.98, Diagnosis = 0.97, Radiation = 0.96, Medication = 0.96.
• Lower entities: Imaging = 0.80, Recurrence Status = 0.84, Comorbidities = 0.89.
• Staging F1 = 0.92; Surgery F1 not shown in Table 3 excerpt but precision 0.994, recall 0.963.
• Biomarkers and medications surpass 0.95 F1.
• Manual evaluation: 94.1% direct approval by oncology data abstractors on a 940,923-datapoint corpus.
• No direct head-to-head comparison against a single baseline system is reported; comparisons are to prior literature upper bounds (e.g., plain GPT-4 ≈30% on complex oncology vignettes in cited work; single-pass systems limited to narrow concept sets).

Limitations

• Evaluated exclusively on melanoma patients; generalizability to other cancer types requires only valueset modification per authors' claim, but is not empirically validated in the paper.
• Development set (n=1,125) was used for prompt engineering and pipeline tuning, risking overfitting to the dataset's institutional distribution (hundreds of sites but all within the Ontada/McKesson network).
• No ablation study quantifying individual contributions of retrieval strategy, pipeline type, or collation logic.
• Imaging entity F1 is notably lower (0.80), attributed implicitly to heterogeneous radiology report formats; no remediation is described.
• Scanned PDF quality is a preprocessing dependency; VLM transcription errors propagate downstream without explicit error-recovery.
• LLM costs and latency at scale (~180 docs/patient × 2,250 patients) are not discussed.
• Ground truth itself may contain abstractor inconsistencies (acknowledged via the manual disagreement-review protocol).

Relevance to Harnesses / Meta-Harnesses

HARMON-E is a direct instance of a domain-specific meta-harness: it orchestrates multiple specialized sub-agents (retrievers, LLM synthesizers, collators) through a configurable, dependency-aware pipeline, mirroring the structural concerns of general meta-harness design — task decomposition, inter-agent dependency ordering, schema validation, and result collation. The four swappable pipeline strategies (Single-Step, Multi-Step, Topical, Sequential) are a concrete realization of adaptive pipeline selection based on task complexity, a pattern increasingly relevant to harness designers targeting heterogeneous data sources. The paper's approach to cross-entity dependency management (topological ordering of collators) and production integration metrics (acceptance rate as a harness-quality proxy) offer transferable engineering lessons for anyone building multi-stage agentic pipelines over unstructured corpora.