Skip to content

MedPAO: A Protocol-Driven Agent for Structuring Medical Reports

🕒 Published (v1): 2025-10-06 09:32 UTC · Source: Arxiv · Venue: MICCAI 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedPAO is a protocol-driven agentic framework for structuring free-text radiology reports into clinically standardized formats (e.g., the ABCDEF chest X-ray protocol) using a Plan-Act-Observe (PAO) loop over a modular toolchain. It achieves F1 of 0.96 on concept categorization and earns a mean expert rating of 4.52/5, outperforming monolithic LLM baselines on both accuracy and structural compliance.

Problem

Free-text radiology reports are inconsistent across clinicians, hindering large-scale data aggregation and downstream predictive modeling. Existing approaches — rule-based systems, supervised ML, and monolithic LLMs — are either brittle, annotation-expensive, or prone to hallucination with no verifiable reasoning chain against established clinical protocols.

Method

MedPAO integrates the Model Context Protocol (MCP) with a Plan-Act-Observe (PAO) loop to orchestrate six specialized tools sequentially: 1. get_concept — fine-tuned MedLlama-8B (LoRA on 200 expert+synthetic samples) extracts medical concepts and source sentences from the raw report. 2. ontology_mapping — maps concepts to SNOMED-CT and RADLEX via BioPortal Annotator API; results cached locally. 3. ontology_filtering — an LLM (DeepSeek-R1-70B) classifies ontology mappings into Primary (pathological finding) vs. Secondary (modifier, location) using few-shot prompting. 4. categorize_concepts — the LLM assigns each concept to one of the six ABCDEF protocol categories, conditioned on the full protocol description plus filtered ontologies. 5. generate_report — the LLM reconstructs protocol-aligned structured findings from categorized concepts and their original source sentences. 6. check_cache — retrieves pre-processed concept mappings to reduce redundant computation.

The PAO loop's Observe state prompts the LLM to verify goal completion after each tool call, either continuing iteration or terminating with the final structured output.

Key Contributions

  • Zero-shot agentic framework grounding report structuring in an explicit clinical protocol (ABCDEF) rather than opaque end-to-end generation.
  • Fine-tuned MedLlama-8B concept extractor trained on 200 samples (40 expert-annotated + 160 synthetic via GPT-4/Gemini/Claude/DeepSeek few-shot) covering 311 unique SNOMED-CT/RADLEX concepts.
  • Modular, verifiable toolchain with ontology-enriched concept context enabling clinically traceable reasoning.
  • Expert radiologist/clinician evaluation protocol using dual metrics (Accuracy Score + Structure Score) on 50 MIMIC-CXR samples.

Results

  • Concept categorization (Table 2): MedPAO agent achieves macro/weighted F1 of 0.94/0.96 and Jaccard of 0.92/0.94, vs. DeepSeek-R1-70B at 0.91/0.94 (\(p=0.019\)) and Qwen2.5-72B at 0.92/0.95 (\(p=0.261\)).
  • Concept extraction (Table 1, @80% fuzzy): MedPAO macro/weighted F1 of 0.86/0.91 vs. DeepSeek-R1-70B at 0.87/0.90 (not statistically significant, \(p=0.361\)); outperforms 7B-class models significantly.
  • Expert evaluation (Table 4): clinician accuracy/structure scores of 4.64/4.59; panel of two radiologists 4.52/4.48 out of 5 on 50 MIMIC-CXR reports.
  • Inference (Table 5): full freestyle-to-structured-report pipeline runs in ~280 s; cache-enabled shortcut reduces to ~66 s.

Limitations

  • Ground-truth dataset is small (200 samples for extraction, 50 for clinical evaluation), limiting statistical power.
  • Evaluation is restricted to chest X-ray reports using the ABCDEF protocol; generalization to other modalities or protocols is claimed but not demonstrated.
  • End-to-end pipeline latency (~280 s) is incompatible with real-time clinical deployment without hardware optimization or further caching.
  • Synthetic training data generated by foundation models may introduce systematic biases from those models into the fine-tuned extractor.
  • The Observe step relies on the LLM's self-assessment of goal completion, which is itself susceptible to the hallucination problem the system aims to solve.

Relevance to Harnesses / Meta-Harnesses

MedPAO is a direct instance of a domain-constrained meta-harness: the PAO loop acts as the orchestrator, and the six MCP tools form a fixed, sequentially composed pipeline whose execution order is determined by an LLM planner rather than hardcoded control flow. The explicit use of MCP as the tool-invocation layer mirrors the server/client decomposition common in general-purpose agent harnesses, but with the added constraint that the protocol schema (ABCDEF) is injected as a first-class input at the categorization step, effectively acting as a declarative task specification. For researchers building or studying meta-harnesses, MedPAO illustrates how embedding a domain schema into the orchestration layer (rather than relying on the LLM's implicit world knowledge) reduces hallucination and improves verifiability — a design pattern directly transferable to other structured-output harnesses.