Skip to content

Prompt, Plan, Extract: Zero-Shot Agentic LLMs Workflows for Lung Pathology Extraction from Clinical Narratives

🕒 Published (v1): 2026-06-18 07:00 UTC · Source: Arxiv · link

Why this paper was selected

Zero-shot agentic workflow harness for clinical NLP; concrete Prompt→Plan→Extract pipeline pattern directly reusable

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper builds a four-node LangGraph agentic workflow (Mapper → Planner → Executor → Compiler) to populate 13 CAP-aligned synoptic fields from lung resection pathology reports in a fully zero-shot regime. Five open-source LLMs are benchmarked against a supervised GatorTron NER-RE pipeline on 53 held-out reports. The best zero-shot model (GPT-OSS-20B) achieves Micro-F1 of 0.893 vs. 0.960 for the supervised baseline, with zero annotation or training cost.

Problem

Lung cancer staging requires extracting structured relational data (tumor size, pathologic stage, margin status, lymph node counts, etc.) from narrative pathology reports. Supervised NER-RE pipelines require expensive manual annotation and suffer cascade failures when upstream entity spans are missed. Zero-shot LLMs without scaffolding yield inconsistent outputs on complex, guideline-governed extraction tasks.

Method

A graph-structured workflow implemented in LangGraph with four sequential nodes:

  1. Mapper — segments the raw report into canonical sections (SPECIMEN, TUMOR, MARGINS, LYMPH NODES, PATHOLOGIC STAGE) without summarization.
  2. Planner — extracts high-level metadata per section (focality hints, neoadjuvant context, margin involvement cues) and builds a task queue; sections planned in parallel.
  3. Executor — fires specialized per-section extraction prompts concurrently, each including the mapped section text, planner metadata, and type constraints (integer counts, float dimensions, null for absent values); an add_attrs catch-all captures unlisted key-value pairs.
  4. Compiler — merges per-task JSON fragments, deduplicates, resolves conflicts, and outputs a single CAP-aligned JSON with 13 fields.

All prompts are held constant across five LLM backends (GPT-OSS-20B, GPT-OSS-120B, Gemma-3-27B-it, Llama-3.3-70B-Instruct, Granite-3.3-8B-Instruct); temperature = 0.0; strictly zero-shot (no exemplars). The supervised baseline uses GatorTron-345M fine-tuned separately for NER (token labeling, 256-token window, 30 epochs, ~8 h A100) and RE (~14 h A100).

Evaluation uses a concept-level framework: outputs are normalized (lowercasing, unit conversion, synonym mapping), deduplicated per field per report, and an "Implicit Null" forgiveness rule neutralizes false-positive penalization for valid negative extractions absent from gold annotations.

Key Contributions

  • A modular four-node zero-shot agentic pipeline for CAP synoptic field population requiring no labeled data.
  • A concept-level, registry-aligned evaluation framework that accounts for clinical deduplication and annotator sparsity, as opposed to strict text-span matching.
  • Systematic comparison of five open-source LLMs (8B–120B parameters) against a supervised NER-RE state-of-the-art on a real EHR corpus (286 reports annotated, 53 held-out test).
  • Demonstration that the framework is model-agnostic (drop-in backend replacement without code changes).

Results

  • Supervised GatorTron NER-RE: Micro-F1 0.960, Precision 0.944, Recall 0.977.
  • GPT-OSS-20B (best zero-shot): Micro-F1 0.893, Precision 0.843, Recall 0.949 — 6.7 pp below supervised.
  • GPT-OSS-120B: F1 0.883; Llama-3.3-70B: F1 0.872; Gemma-3-27B: F1 0.831; Granite-3.3-8B: F1 0.834.
  • All five zero-shot models achieve Global Recall > 0.92.
  • Granite-3.3-8B achieves F1 0.994 on Pathologic Stage (vs. supervised 0.970), outperforming the baseline on this synthesis-heavy field.
  • Visceral Pleural Invasion remains hardest for LLMs (F1 0.308–0.348) vs. supervised (1.000); Tumor Size also lags (best LLM 0.849 vs. 0.931).
  • Median inference latency: 24.65 s/report (GPT-OSS-20B) to 35.71 s/report (GPT-OSS-120B); supervised required ~22 h GPU training + 286-document annotation corpus.

Limitations

  • Prompts were manually engineered and not sensitivity-tested for semantic variation; robustness to phrasing shifts is unquantified.
  • Only 53 test reports from a single institution (UF Health, 2018–2022); generalizability to other health systems or reporting styles untested.
  • Gold standard has annotator sparsity (omitted negatives), requiring ad hoc forgiveness rules that may mask true precision differences.
  • VPI and LVI fields show severely degraded LLM precision due to the implicit-null phenomenon, unresolved by current evaluation design.
  • No formal ablation of individual workflow nodes (e.g., Mapper vs. no-Mapper) to isolate contribution of each stage.
  • Temperature 0.0 and max 2 retries may not fully address formatting failures for weaker models.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a task-specific agentic harness: LangGraph is used as the orchestration substrate to wire four specialized LLM calls into a deterministic DAG, with parallel fan-out in the Planner and Executor nodes — a micro-scale meta-harness pattern. The model-agnostic backend swap (changing only the endpoint, not the graph logic) is exactly the kind of harness/meta-harness separation of concerns that generalizes across LLM providers. The evaluation framework itself functions as a harness component: it normalizes, deduplicates, and scores structured outputs independent of the extraction engine, mirroring how meta-harnesses wrap evaluation pipelines around interchangeable agent backends. The paper provides a concrete, measurable case study of how harness structure (node decomposition, parallel execution, compiler aggregation) compensates for raw zero-shot model inconsistency.