Skip to content

INFORM-CT: INtegrating LLMs and VLMs FOR Incidental Findings Management in Abdominal CT

🕒 Published (v1): 2025-12-10 23:28 UTC · Source: Arxiv · Venue: MIDL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

INFORM-CT is an agentic plan-and-execute framework that automates incidental findings management in abdominal CT by having an LLM (Claude 3.5) generate Python programs from parsed clinical guideline PDFs, which are then executed using VLM and segmentation base functions. It substantially outperforms pure VLM baselines on a multi-organ benchmark (liver, kidney, pancreas) while producing interpretable decision trajectories aligned with ACR guidelines.

Problem

No prior method provides a general-purpose, guideline-adherent pipeline for detecting and classifying incidental findings across multiple abdominal organs in CT. Existing deep learning approaches target single pathology types or organs, and pure VLMs (e.g., MERLIN) lack the procedural reasoning and quantitative image analysis (size, HU intensity, border thickness) required to follow multi-step clinical decision trees.

Method

The framework has three stages: 1. Guideline parsing: GPT-4o + LangChain converts ACR guideline PDFs (text, figures, tables) into structured JSON decision trees. 2. Planner (ReAct agent on Claude 3.5): Given the JSON tree and a library of base-function API descriptions, the planner generates a Python program encoding the full clinical decision logic via multi-turn iterative refinement (draft → execute → feedback → regenerate until a STOP criterion on syntactic and semantic validity is met). 3. Executor: Runs the generated program, invoking base functions: organ/tumor segmentation (TotalSegmentator, nnUNet), morphometric measurements (diameter in mm/cm, Hausdorff border thickness, Hounsfield unit intensity), and a VLM labeler (MERLIN 3D CLIP) for higher-level fine-grained attributes (e.g., "suspicious," "flash-filling"). Outputs a patient-level recommendation with an explainable decision path.

Ground-truth labels are extracted from paired radiology reports using GPT-4o path matching over the decision tree.

Key Contributions

  • First end-to-end agentic pipeline for incidental findings management spanning multiple abdominal organs (liver, pancreas, kidney), derived automatically from guideline PDFs.
  • Plan-and-execute code generation that translates complex clinical decision trees into executable Python with quantitative image processing primitives—substantially more complex programs than prior VQA-style code-generation approaches.
  • A multi-organ CT benchmark (168/188/98 scans for liver/pancreas/kidney) with ground-truth recommendations extracted from radiology reports via LLM-based decision-path matching.

Results

  • Recommendation accuracy vs. Pure MERLIN baseline (accuracy % / weighted F1):
  • Liver: 63.09 / 61.38 vs. 12.5 / 14.33 (random: 10.0)
  • Kidney: 60.0 / 61.0 vs. 48.8 / 14.0 (random: 16.67)
  • Pancreas: 41.48 / 46.32 vs. 11.17 / 12.96 (random: 7.14)
  • Explanation accuracy (liver): INFORM-CT 54.76% vs. Pure MERLIN 4.76% on decision trajectory matching.
  • Ablation (liver): Removing segmentation base functions and replacing with MERLIN drops recommendation accuracy from 63.09% to 20.45%, confirming segmentation is the critical component.

Limitations

  • Performance is bounded by the quality of underlying segmentation models; no strong universal lesion segmentation models exist for all abdominal organs.
  • VLM labeler (MERLIN) can misclassify fine-grained imaging features (e.g., labeling benign hepatic masses as "suspicious"), producing downstream recommendation errors even when segmentation is correct.
  • Benchmark uses an internal dataset; no public dataset evaluation reported.
  • Ground-truth extraction via LLM path-matching introduces its own error source.
  • Only three organs tested; generalization to the full abdominal organ set is undemonstrated.

Relevance to Agentic AI / LLM Agents

INFORM-CT is a concrete application of the plan-and-execute agentic pattern to a high-stakes domain where the agent must translate structured external knowledge (PDF guidelines → JSON decision trees) into executable programs that orchestrate heterogeneous specialist tools (segmentation models, VLMs, image processing routines). It demonstrates that ReAct-style iterative code generation with domain-specific base-function libraries can dramatically outperform monolithic VLM inference when tasks require multi-step quantitative reasoning. The ablation result—that neither the LLM program nor the VLM alone suffices, but their composition does—is directly relevant to debates about when tool-augmented agents exceed pure foundation model capabilities. The work also exemplifies guideline-grounded agentic behavior, a pattern of growing importance for deploying LLM agents in regulated domains.