Skip to content

TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; interleaved multimodal CoT enables traceable clinical tumor analysis

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TumorChain is a multimodal interleaved chain-of-thought (CoT) framework for 3D CT-based clinical tumor analysis that couples organ segmentation, auxiliary classification, and an LLM in an iterative loop to produce traceable "findings → impression → pathology" reasoning chains. The authors also release TumorCoT-1.5M, a 1.5M VQA dataset with step-aligned CoT annotations across five digestive organs, and propose TumorChain-Eval (CoTe), a triplet-based stepwise evaluation metric. The system demonstrates consistent gains over general and medical VLM baselines on both an in-house benchmark and the public DeepTumorVQA benchmark.

Problem

Existing medical LVLMs suffer from three compounding gaps: (i) they are optimized for broad clinical tasks (report generation, coarse classification) and fail to connect radiological findings to pathology-level endpoints such as TNM staging and nodal metastasis; (ii) available tumor-centric datasets (e.g., CT-RATE, 3D-RAD) use constrained multiple-choice or short-text formats without multi-granular or stepwise reasoning annotations; and (iii) most models process 2D images with single-pass inference, which is structurally inadequate for the multi-stage clinical inference required in 3D radiologic scenarios.

Method

TumorChain decomposes into two coupled subsystems:

Global & Local Visual Alignment. A 3D vision encoder \(E_v\) encodes CT volume \(V_{ct} \in \mathbb{R}^{H \times W \times D}\) into global tokens \(\tau_g = P(E_v(V_{ct}))\). An organ segmentation expert \(\text{Seg}(\cdot)\) (TotalSegmentator) produces full organ masks \(M_{\text{organ}}\); the task-relevant organ mask \(M_{\text{task}} = \Lambda(M_{\text{organ}}, T_{\text{task}})\) is used to extract local tokens \(\tau_l = \Gamma(\tau_g, M_{\text{task}})\). An auxiliary classification model \(\text{Cls}(\tau_l)\) outputs normal/abnormal logits to sharpen the encoder's discriminative learning via a joint loss: $\(\mathcal{L}_{\text{total}} = -\frac{1}{N}\sum_{n=0}^N \log P(r_n | \tau_{\text{in}}, r_{1:n-1}) - \alpha \cdot \frac{1}{M}\sum_{m=0}^M \hat{y} P(y|\tau_l).\)$

Organ-guided Iterative Interleaved Reasoning (IIR). At each step \(i\), the LLM generates \(R^i_{\text{cot}}\); keywords in the output identify new relevant organs, whose local tokens are extracted and prepended as augmented prompts (e.g., "The question requires greater attention to [organ]"). The combined input for the next step is \(\tau^{i+1}_{\text{in}} = [\tau_g, T_{\text{task}}, R^i_{\text{cot}}, T^i, \tau^i_l]\), iterating until no new ROIs emerge.

Data Engine. A multi-agent pipeline converts raw radiology and pathology reports into CoT-VQA pairs using: a Segmentation Expert and Feature Extractor for structured feature extraction; a diagnostic knowledge graph encoding organ sub-structures and clinical relationships; and three LLM agents—CoT Reasoner (GPT-4o-mini), Logic Calibrator (Claude 3.5-Haiku), and Summarizer (GPT-5-mini)—that generate, validate, and aggregate reasoning chains.

TumorChain-Eval (CoTe). Ground-truth and predicted CoT chains are parsed into subject–relation–object triplets across three levels—Finding Chain (\(S_{FC}\)), Impression Chain (\(S_{IC}\)), Long Reasoning Chain (\(S_{LRC}\))—and scored by GPT-4 according to clinical scoring rubrics. The final metric is a weighted sum \(\text{CoTe} = w_1 S_{FC} + w_2 S_{IC} + w_3 S_{LRC}\).

Key Contributions

  • TumorCoT-1.5M: 1,497,818 CoT-VQA instruction pairs covering four task types (localization, lesion attribute analysis, TNM stage prediction, CoT report generation) across five digestive organs; built from 41,059 CT volumes and 10,708 radiology reports at 9:1 patient-level train/test split.
  • TumorChain-Eval (CoTe): Triplet-based, stepwise evaluation protocol measuring both final accuracy and logical consistency of multi-step CoT chains, addressing the inability of ROUGE-L/CIDEr to capture clinical reasoning correctness.
  • IIR architecture: A dynamic, iterative multimodal inference loop that progressively fetches organ-specific visual evidence guided by the LLM's own intermediate outputs.
  • Hybrid-model Collaborative Optimization (HCO): Joint training of segmentation expert, auxiliary classifier, and LLM enabling organ-level global–local alignment for 3D CT data.
  • Clinical pipeline formalization: Operationalization of the radiology findings → impressions → pathology prediction trajectory as a standardized ML benchmark.

Results

The full numerical results table is truncated in the provided text; only partial baseline rows are visible. Based on what the paper reports:

  • TumorChain (3B and 7B variants, built on Qwen2.5-VL) achieves consistent gains over all baselines on the TumorCoT-1.5M test set across lesion detection, impression quality, and pathology classification tasks.
  • Baselines include seven general-domain LVLMs (Claude 3, Gemini 2.0-Flash, GPT-5-Mini, Qwen2.5-VL-7B, InternVL-2.5-8B, MiniCPM-V4.5-9B, LLaVA-CoT-11B), three 2D medical LVLMs (HealthGPT-3.8B, Lingshu-7B, MedVLM-R1-2B), and two 3D medical LVLMs (RadFM, M3D-Phi-3).
  • TumorChain generalizes to the public DeepTumorVQA benchmark without benchmark-specific tuning.
  • Ablations confirm that both interleaved reasoning (IIR) and clinical CoT supervision are necessary for the observed gains.
  • From the partial table: general-domain models show widely varying position-detection scores (e.g., Claude3-Haiku: Organ Pos. 32.61, Tumor Pos. 33.05; GPT-5-Mini: Organ Pos. 44.18, Tumor Pos. 37.73), underscoring the task difficulty for non-specialized models.

Limitations

  • TumorCoT-1.5M CoT annotations are generated by LLM agents (GPT-4o-mini, Claude 3.5-Haiku, GPT-5-mini) from text reports; image-grounding of individual chain steps is not independently verified by radiologists at scale.
  • The benchmark and model are restricted to five digestive organs; generalization to other cancer types or imaging modalities (MRI, PET) is undemonstrated.
  • TumorChain-Eval (CoTe) relies on GPT-4 as a scoring oracle, introducing potential hallucination or calibration biases in the evaluator itself.
  • IIR iteration count and termination are governed by LLM keyword extraction heuristics, which may miss relevant organs or over-expand to spurious ROIs in noisy inputs.
  • Results table in the available text is truncated, preventing independent verification of specific numerical claims.

Relevance to Vision-Language Models

TumorChain instantiates a concrete architectural pattern for dynamic visual grounding in VLMs: rather than fixing the set of visual tokens at inference time, the LLM's intermediate text output controls which image regions are fetched and re-injected, generalizing single-pass cross-attention to an iterative loop. This is directly relevant to the broader VLM agenda of reducing hallucination and improving faithfulness on visually complex inputs. The paper also demonstrates that domain-specific CoT supervision—structured as a clinical reasoning chain rather than generic instruction tuning—is a key lever for closing the gap between general-purpose LVLMs and specialist-grade performance in 3D medical imaging. The TumorCoT-1.5M dataset and CoTe metric provide a new evaluation infrastructure that could inform how VLM benchmarks in other high-stakes visual domains are constructed.