Skip to content

MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology

🕒 Published (v1): 2025-11-25 16:56 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MTBBench is an oncology benchmark that simulates Molecular Tumor Board (MTB) workflows, requiring AI agents to reason over multimodal (H&E, IHC, hematology, genomics) and longitudinal patient data across sequential clinical decision points. All baselines struggle—best models reach ~69% on multimodal tasks and ~67% on longitudinal—with outcome/recurrence prediction near chance. Augmenting agents with domain-specific foundation model tools yields gains of up to 9.0% (multimodal) and 11.2% (longitudinal).

Problem

Existing clinical benchmarks evaluate static, single-turn, unimodal question-answering and cannot capture the complexity of real MTB workflows, which demand: (1) integration of heterogeneous modalities (pathology images, lab values, genomic data), (2) temporally sequential reasoning as patient histories evolve, and (3) active multi-turn information retrieval rather than passive input consumption. No prior benchmark jointly addresses multimodality, longitudinality, and interactive/multi-agent decision-making in a clinician-validated oncology setting.

Method

MTBBench comprises two tracks: - MTBBench-Multimodal: 390 MC/T-F Q&A pairs across 26 head-and-neck cancer patients (HANCOCK dataset), spanning digital pathology (H&E, IHC), hematology, and outcome/recurrence prediction across three sequential clinical stages. - MTBBench-Longitudinal: 183 manually constructed Q&A pairs across 40 MSK-CHORD patients, with clinicogenomic timelines segmented into decision-relevant timepoints for outcome, progression, and recurrence reasoning.

Agents operate in a multi-turn agentic loop: at each turn t, the agent receives a query and a file list, must actively request files (non-persistent across turns), and accumulates a reasoning history. An optional tool layer exposes: (a) CONCH (H&E VLM via image-text cosine similarity), (b) a custom UNI2+ABMIL IHC quantification tool, (c) PubMed retrieval reranked via BAAI-bge-reranker-v2-m3, and (d) DrugBank pharmacological metadata. Expert annotations are validated via a custom companion web app.

Key Contributions

  • First benchmark to jointly address multimodality, longitudinality, and interactive multi-agent evaluation in clinician-validated precision oncology.
  • Systematic evaluation of 10+ open- and closed-source LLMs/VLMs (Gemma-3, GPT-4o, o4-mini, InternVL3, LLaMA, Mistral, Qwen2.5/3) under tool-free and tool-augmented conditions.
  • Modular agentic framework integrating domain-specific pathology foundation models (CONCH, UNI2) and biomedical knowledge tools as callable APIs.
  • Expert-validated dataset, companion annotation app, agent logs, and open-source release.

Results

Baseline (no tools), MTBBench-Multimodal: - Best overall: internvl3-78b at 69.1% (vs. GPT-4o at 66.7%) - Digital pathology: 61.1%–82.8% across models; no consistent scaling benefit with model size - Hematology: 42.1%–65.6% - Outcome/recurrence: near chance (~50–69%), most difficult task - Cross-model accuracy gap: up to 36.7% (digital pathology), 34.6% (outcome/recurrence)

Baseline (no tools), MTBBench-Longitudinal: - Best: qwen3-32b at 67.0% overall; 83.0% on outcome prediction - Recurrence and progression remain near chance (~47–57%)

With tools: - Multimodal digital pathology: up to +9.0% (gemma-3-12b, qwen25-7b) - Longitudinal progression/recurrence: >5% gains in select models - Outcome prediction (already strong baseline) sees incremental gains

Key finding: Number of files/modalities accessed is a stronger predictor of accuracy than model scale.

Limitations

  • Benchmark is offline and controlled; agents are not tested in interactive real-world clinical workflows with ambiguous or incomplete inputs.
  • MTBBench-Longitudinal lacks specialized foundation models for temporal clinical reasoning; only general-purpose tools (PubMed, DrugBank) are available, limiting gains.
  • Small patient cohorts: 26 patients (multimodal), 40 patients (longitudinal); wide confidence intervals (±6–14%) reflect this.
  • Q&A generation for multimodal track used GPT-4o with only select expert review, not full clinician validation of all 390 pairs.
  • Head-and-neck cancer focus limits generalizability to other oncology domains.
  • No evaluation of agent strategies (e.g., file selection policies) beyond accuracy metrics.

Relevance to Agentic AI / LLM Agents

MTBBench directly advances the frontier of tool-augmented, multi-turn agentic evaluation by instantiating a realistic, high-stakes domain where passive LLM capabilities are insufficient and active information gathering, tool invocation, and longitudinal memory are essential. The finding that file access breadth outpredicts model scale has direct implications for agent design—loop structure and retrieval strategy matter more than parameter count. The integration of domain-specific foundation models as callable tool APIs (CONCH, UNI2) is a concrete instantiation of the "specialist tool" paradigm increasingly central to medical and scientific agents. For researchers tracking agentic AI, MTBBench provides a rare clinician-grounded testbed where hallucination, conflicting-evidence reconciliation, and temporal reasoning failures are directly measurable.