Skip to content

Radiologist Copilot: An Agentic Framework Orchestrating Specialized Tools for Reliable Radiology Reporting

🕒 Published (v1): 2025-12-02 14:25 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Radiologist Copilot is an agentic framework built atop OctoTools that orchestrates four specialized tools—Segmentator, Analyzer, Report Composer, Quality Controller—to automate the complete 3D CT radiology reporting workflow, not merely isolated report generation. It combines region analysis planning, strategic template selection, and feedback-driven quality control. On liver reporting (AMOS-MM), it substantially outperforms all 3D medical VLM baselines across NLG and clinical efficacy metrics.

Problem

Existing automated radiology reporting approaches—including 3D medical VLMs and prior agentic models—address only isolated report generation, omitting clinically necessary stages: region-of-interest localization, structured image analysis, template selection, and quality assurance. A single-pass model cannot handle this multi-stage, planning-intensive workflow.

Method

Radiologist Copilot frames radiology reporting as an LLM-driven agentic loop (Eq. 1: \(a_t \sim \pi_{\theta_l}(a \mid h_{t-1})\)) over a curated tool library:

  1. Segmentator Tool: TotalSegmentator extracts organ mask \(M_\text{organ}\) and lesion mask \(M_\text{lesion}\) from the 3D CT \(I \in \mathbb{R}^{C \times H \times W \times D}\).
  2. Analyzer Tool (Region Analysis Planning): Identifies region-specific analysis items (anatomical structures + clinically significant features); conditionally adds lesion characteristics if \(|M_\text{lesion}| > 0\) (Eq. 2). Items are formatted as prompts \(\phi(\mathcal{I})\) for a 3D medical VLM (Hulu-Med).
  3. Report Composer Tool (Strategic Template Selection): Candidate templates \(\mathcal{C}\) are derived by K-means clustering of historical reports (BioBERT embeddings); the LLM selects \(C^* = \arg\max_{C_j \in \mathcal{C}} p(C_j \mid \text{Ana}, \text{feedback}_\text{opt})\) (Eq. 3) and generates \(R_\text{gen}\) (Eq. 4). Output includes findings, impression, and key CT slice references (Eq. 5).
  4. Quality Controller Tool: LLM assesses format, content, and language; if \(\text{QA}(R_\text{gen}) = 0\), feedback triggers iterative adaptive refinement (Eq. 6) until the report is qualified.

The backbone is Qwen3-32B; the system is training-free and implemented on OctoTools, max 10 steps.

Key Contributions

  • An end-to-end agentic framework for the full radiology reporting workflow (localization → analysis → template selection → composition → quality control), contrasting with prior single-tool agentic methods.
  • Region analysis planning: adaptive, structure-aware prompting of a 3D VLM conditioned on segmentation outputs and lesion presence.
  • Strategic template selection via unsupervised clustering of historical reports, enabling standardized, case-adapted report writing.
  • Feedback-driven iterative quality control that detects content inconsistency, spelling errors, and format violations.
  • Dual evaluation protocol: task-level NLG+CE metrics and agent-level LLM-as-a-Judge (GPT-5.2) across four dimensions.

Results

All results on the AMOS-MM liver reporting validation set (367 CT scans):

  • BLEU-1: 0.4025 vs. best VLM baseline CT-CHAT 0.2440 (+65%)
  • ROUGE-L: 0.3222 vs. CT-CHAT 0.2012 (+60%)
  • METEOR: 0.4560 vs. CT-CHAT 0.2599 (+75%)
  • BERTScore: 0.7024 vs. CT-CHAT 0.6127
  • F1-RadGraph: 0.2585 vs. Merlin 0.1617 (+60%)
  • GREEN: 0.4379 vs. Hulu-Med 0.2163 (+102%)
  • Agent-level (LLM-as-a-Judge, 1–5): majority scores 4–5 across all four dimensions (Problem Analysis, Action Planning, Action Execution, Overall Workflow)
  • Ablation: removing region analysis planning or template selection degrades all primary metrics; removing quality control shows smaller but non-zero impact

Limitations

  • Evaluated on a single organ (liver) from one dataset (AMOS-MM); generalizability to other organs/modalities is unvalidated.
  • Quality Controller relies on an LLM judge, inheriting that model's hallucination and inconsistency risks.
  • Template clustering is unsupervised and dataset-specific; templates must be re-derived for new institutions or report styles.
  • Agent-level evaluation uses GPT-5.2 as judge—an unreleased model at the time of the arxiv submission, raising reproducibility concerns.
  • Code not yet released (pending acceptance).
  • Maximum step count (10) may be insufficient for complex multi-lesion cases; no analysis of failure modes when the step budget is exhausted.

Relevance to Harnesses / Meta-Harnesses

Radiologist Copilot is a concrete instantiation of a domain-specific meta-harness: an LLM orchestrator that dynamically selects and sequences specialized sub-tools (segmentation model, 3D VLM, template retriever, quality judge) rather than encoding any domain logic itself. The feedback-driven adaptive refinement loop (Eq. 6) is precisely the self-correction pattern central to meta-harness design, where a supervisor module gates output quality and re-invokes downstream tools. Built on OctoTools, it demonstrates how a general agentic scaffold can be specialized into a reliable, multi-stage pipeline with domain-specific quality gates—a directly transferable architectural pattern for anyone building orchestration harnesses over heterogeneous tool libraries.