Skip to content

MonteRET: AI Agent Enhancing Multimodal LLMs with Multi-granularity Knowledge Retrieval for Chest CT Report Generation

🕒 Published (v1): 2026-07-15 18:17 UTC · Source: Arxiv · link

Why this paper was selected

Multi-granularity RAG agent for chest CT reports; retrospective clinical evaluation included

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MonteRET is a retrieval-augmented generation framework for chest CT report generation that fuses global volumetric features with region-level anatomical representations across 10 predefined anatomical zones. It retrieves clinically relevant prior reports using both condition classification and region-level vision–language alignment, then refines the draft report via an LLM agent using chain-of-thought and in-context learning prompting. On RadGenome-ChestCT it outperforms all tested state-of-the-art methods in both language quality and clinical completeness, with the largest gains in recall of medical findings.

Problem

Automated chest CT report generation fails on two fronts: (1) models relying on global volumetric representations miss subtle localized findings (e.g., apical nodules, mild pleural effusion), and (2) single-round LLM generation produces factually inconsistent outputs that hallucinate or omit findings due to insufficient grounding in external medical knowledge. Existing methods do not jointly address multi-granularity anatomical modeling and knowledge-augmented retrieval in a single framework.

Method

MonteRET comprises three sequentially coupled modules:

  1. Report Generation Module: A ViT3D encoder extracts global CT features; a segmentation module generates 10 anatomical region masks (abdomen, bone, breast, heart, esophagus, lung, mediastinum, pleura, thyroid, trachea/bronchi) to produce region-level feature embeddings. Random region masking during training forces the model to rely on both global and local features. These multi-granularity representations condition a fine-tuned LLaMA-2-7B-Chat decoder to produce a preliminary report. A vision–language alignment head produces region-level retrieval tokens aligned with paired report sentences (192,242 region-sentence pairs).

  2. Knowledge-Augmented Retrieval (KAR) Module: Two complementary strategies are combined:

  3. Classification-based retrieval: A multi-label classifier predicts which of 18 medical conditions are present; the knowledge base is queried for reports containing those predicted conditions.

  4. Image-to-text (I2T) retrieval: The region-level retrieval token retrieves semantically aligned CT reports from the knowledge base via vision–language similarity. Top-\(K\) retrieved reports are passed as context.

  5. Report Refinement Module: A ChatGPT-based agent rewrites the preliminary report conditioned on retrieved evidence using prompt engineering: Chain-of-Thought (intermediate reasoning steps), In-Context Learning (retrieved reports as few-shot examples), and constraint instructions (majority-vote across retrieved reports determines condition presence).

Training used 24,128 CT scan–report pairs from RadGenome-ChestCT (built on CT-RATE).

Key Contributions

  • Multi-granularity anatomical encoder combining global ViT3D features with 10 anatomically masked region representations
  • Dual retrieval strategy fusing classification-based (condition-guided) and image-to-text (VL alignment–based) retrieval, shown by ablation to be complementary
  • Region-level vision–language alignment module generating retrieval tokens for anatomically grounded evidence selection
  • Random region masking training objective for robustness to local feature dropout
  • LLM-based report refinement agent with CoT + ICL + constraint prompting
  • Evaluation across BLEU, ROUGE-L, METEOR, CIDEr, BERTScore, GPT-4o judge, 4-resident blinded review, and 18-condition clinical efficacy on both internal and external cohorts

Results

vs. matched baseline (RadGenome-ChestCT, \(n=1{,}564\)): - BLEU-4: 0.252 (+3.3 pts); ROUGE-L: 0.379 (+2.99 pts); METEOR: 0.454 (+4.06 pts) (all \(p < 0.0001\)) - BERTScore: 0.426 (+1.38 pts, \(p < 0.0001\)) - Clinical recall: 0.406 (+33.32 pts, \(p < 0.0001\)); F1: 0.420; Precision: 0.435

vs. Reg2RG (best SOTA, RadGenome-ChestCT): - BLEU-4 +0.28 pts, ROUGE-L +1.20 pts, METEOR +1.29 pts - Clinical recall +22.51 pts, F1 +16.68 pts

External cohort (NYP/WCM, \(n=82\), zero-shot transfer): - BLEU-4 +3.16 pts, ROUGE-L +4.25 pts, METEOR +3.02 pts, BERTScore +5.64 pts vs. baseline (all \(p < 0.0001\)) - Clinical efficacy (precision/recall/F1) showed limited improvement on external cohort

Human expert evaluation (50 blinded cases): - Average Error Severity: MonteRET 0.696–1.174 vs. baseline 1.304–1.913 across four radiology residents (all \(p < 0.05\))

Ablation: - Removing KAR: recall \(-25.6\) pts, F1 \(-20.1\) pts (largest single-component impact) - Removing VL alignment: moderate degradation in both language quality and clinical efficacy - Classification-based retrieval alone outperforms both I2I and I2T retrieval alone; full MonteRET (I2T + classification) best

Limitations

  • Cross-institution clinical efficacy generalization is weak: external NYP/WCM cohort showed strong text-similarity gains but limited improvement in condition-level precision/recall/F1, attributed to institutional style shift and absence of in-domain retrieval examples
  • Condition-level F1 ranges from 0.104 to 0.700; bronchiectasis remains particularly low, likely due to subtle imaging characteristics and low training prevalence
  • Class imbalance in RadGenome-ChestCT training set (dominant lung-related patterns) constrains performance on rare region–condition pairs despite KAR mitigation
  • External refinement agent (ChatGPT) introduces API dependency, cost, latency, and reproducibility concerns
  • External cohort is small (\(n=82\)); statistical power for clinical efficacy analysis is limited

Relevance to Vision-Language Models

MonteRET demonstrates a concrete instantiation of region-level VL alignment for volumetric 3D medical imaging, directly extending the VLM paradigm from 2D images to CT volumes by learning retrieval tokens that align anatomically masked visual features with paired report text. The ablation isolates the contribution of VL alignment as a distinct gain over pure visual similarity, showing that semantic image-to-text grounding is necessary for clinical completeness beyond what image-to-image similarity can provide. The framework also illustrates how retrieval-augmented generation can be composed with VL alignment to reduce LLM hallucination in high-stakes medical settings—a pattern broadly applicable to VLMs in any domain requiring factual grounding. The cross-institution generalization gap for clinical metrics (vs. text metrics) highlights a key open challenge: VL alignment modules trained on one institution's reporting style do not seamlessly transfer when retrieval knowledge bases are out-of-domain.