Skip to content

Multi-Agent Intelligence for Multidisciplinary Decision-Making in Gastrointestinal Oncology

🕒 Published (v1): 2025-12-09 14:56 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A hierarchical multi-agent framework emulates human Multidisciplinary Team (MDT) oncology workflows by routing endoscopy, radiology, laboratory, and clinical text inputs through dedicated specialist agents before aggregation by a central MDT-Core agent. Applied to a multi-institutional GI oncology dataset of 2,174 cases, it achieves a composite expert score of 4.60/5.00 versus 3.76 for a monolithic baseline. The key architectural insight is that task decomposition with explicit cross-modal conflict detection suppresses hallucination and context dilution.

Problem

Monolithic MLLMs processing heterogeneous patient records (endoscopic images, CT/MRI reports, lab values, EMRs) suffer from context dilution—key modality-specific signals are lost in long token sequences—and lack structured mechanisms to resolve inter-modality conflicts, leading to hallucinated or contraindicated treatment recommendations in GI oncology MDT scenarios.

Method

Five agents are composed hierarchically: four parallel specialist agents (Text, Endoscopy, Radiology, Laboratory) each map their raw modality input x_m to a structured intermediate reasoning state h_m, then a central MDT-Core agent ingests the concatenated evidence container H and generates the final diagnostic report.

  • Text Agent (Qwen-3 32B, SFT): extracts structured summaries from EMR narratives.
  • Endoscopy Agent (InternS1 Mini, VQA-SFT on 1,032 patients / 50k+ image-caption pairs): answers clinically framed multi-question probes about lesion morphology, size, invasion depth.
  • Radiology Agent (Qwen-3 32B, SFT): converts CT/MRI narrative reports to structured TNM staging.
  • Laboratory Agent (Qwen-3 32B, SFT): flags abnormal biomarkers (CEA, CA19-9) and interprets clinical significance.
  • MDT-Core Agent (Qwen-3 32B, SFT on concatenated specialist outputs → real expert MDT reports): performs cross-modal conflict detection before generation—if endoscopy and radiology staging differ by >1 level, a Staging Discrepancy Detected flag is injected and the Core recommends pathological confirmation; rule-based threshold validation additionally screens for treatment contraindications.

SFT data for specialist agents was bootstrapped via knowledge distillation: a Teacher LLM reverse-decomposed verified MDT reports into modality-specific reasoning chains, with 10% expert audit for quality control.

Key Contributions

  • Hierarchical multi-agent architecture that mirrors clinical MDT division of labor, explicitly preventing context dilution by confining each modality to a dedicated agent context window.
  • VQA-based Endoscopy Agent that converts raw endoscopic image sequences into structured, clinically actionable textual evidence for downstream reasoning.
  • Explicit rule-based cross-modal conflict detection module with staged intervention (flag + defer to pathology rather than force consensus).
  • Multi-institutional GI oncology dataset (2,174 cases; gastric, colorectal, esophageal) with full multimodal annotations and expert-verified MDT conclusions used for training and evaluation.
  • Dual evaluation protocol: 3-oncologist blind panel + GPT-4-Turbo LLM-as-Judge pairwise comparison for scalable coverage.

Results

  • Composite expert score: 4.60/5.00 (multi-agent) vs. 3.76/5.00 (monolithic Qwen-3-32B baseline); improvement of +0.84.
  • Largest gains per dimension: Reasoning Logic +1.17, Diagnostic Comprehensiveness +0.97, Medical Accuracy +0.86.
  • Ablation—removing each agent degrades composite score: w/o Text Agent (EMR) → 3.65, w/o Radiology Agent → 3.88, w/o Endoscopy Agent → 4.18, w/o Laboratory Agent → 4.32, w/o Conflict Detection → 4.35.
  • VQA paradigm outperforms free-form captioning and slot-filling for endoscopy (qualitative; no numeric score reported).
  • LLM-as-Judge (GPT-4-Turbo) pairwise comparisons correlated strongly with human expert scores; consistently preferred multi-agent outputs on evidence integration and factual consistency.
  • Inference latency reported as under one minute per case with parallel agent execution.

Limitations

  • Radiology Agent operates on textual CT/MRI reports rather than raw DICOM volumes; inherits radiologist transcription errors.
  • Dataset geographically localized to two Shanghai hospitals; cross-population generalizability unvalidated; external cohort validation ongoing but incomplete.
  • Static feed-forward inference: Core cannot issue follow-up queries to specialist agents mid-reasoning (no iterative debate loop).
  • Expert evaluation conducted by sampling test cases, not full 500-case test set; exact sample size for human review not specified.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete domain-specific instance of a meta-harness pattern: a coordinator agent (MDT-Core) orchestrates a fixed set of specialist sub-agents, each with a defined input contract and structured output schema (h_m states), then aggregates results with deterministic conflict-detection rules before final synthesis. The SFT data construction pipeline—where a Teacher LLM reverse-decomposes ground-truth outputs into per-agent training targets—is directly analogous to harness bootstrapping strategies where a meta-level process generates training signal for constituent agents. The explicit synchronization logic (Algorithm 1) and modality-priority concatenation are engineering primitives relevant to anyone designing orchestration layers that must handle heterogeneous, potentially conflicting sub-agent outputs at inference time.