Skip to content

MTBBench: A Multimodal Sequential Clinical Decision-Making Benchmark in Oncology

🕒 Published (v1): 2025-11-25 16:56 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MTBBench is a clinician-validated benchmark that evaluates AI agents on longitudinal, multimodal oncology reasoning simulating Molecular Tumor Board (MTB) workflows. It pairs the benchmark with an agentic framework that exposes domain-specific foundation models (pathology, IHC, literature) as callable tools, yielding gains of up to 9.0% (multimodal) and 11.2% (longitudinal) over baseline LLMs. Even top models score near chance on outcome/recurrence prediction, underscoring the gap between static QA benchmarks and real clinical complexity.

Problem

Existing clinical AI benchmarks assess unimodal, single-turn, static question-answering — failing to capture (1) sequential evidence accumulation across time, (2) heterogeneous data modalities (H&E, IHC, hematology, genomics), and (3) multi-agent decision-making workflows like MTBs where agents must actively retrieve, reconcile, and update hypotheses. No prior benchmark jointly addresses multimodality, longitudinality, and interactive/agentic evaluation in a clinician-validated oncology setting.

Method

MTBBench is organized into two tracks:

MTBBench-Multimodal: 390 MC/T-F question–answer pairs across 26 head-and-neck cancer patients (HANCOCK dataset), covering H&E, IHC, hematology, and outcome/recurrence prediction, unfolding in three sequential clinical stages.

MTBBench-Longitudinal: 183 manually constructed Q&A pairs across 40 cancer patients (MSK-CHORD dataset) with clinicogenomic timelines, requiring reasoning over temporally segmented events (diagnosis → treatment → recurrence/progression).

The agentic framework formulates decision-making as a multi-turn process: at turn \(t\), the agent receives query \(q_t\) and file set \(F_t\); it issues retrieval request \(R_t \subseteq F_t\) (non-persistent across turns), accumulates reasoning history \(h_t\), and maintains an evolving timeline \(T_t = \bigcup_{i=1}^{t} \tau_i\). Domain-specific foundation models are exposed as callable tools: - CONCH (1.17M H&E image–caption pairs): image–text embedding similarity for H&E classification. - UNI2 + ABMIL: 256×256 patches embedded to 1536-dim vectors, aggregated via attention-based MIL to regress IHC staining percentage. - PubMed retrieval: top-30 articles retrieved via NL query, reranked with BAAI-bge-reranker-v2-m3 to top-3 abstracts. - DrugBank: drug mentions in timelines auto-linked to mechanism/indication metadata.

Expert annotation is conducted through a companion web app enabling clinician review of full-resolution slides and per-question feedback.

Key Contributions

  • First benchmark simultaneously covering multimodality, longitudinality, interactive/multi-agent structure, and clinician validation in precision oncology.
  • 573 expert-curated Q&A pairs spanning digital pathology (H&E + IHC), hematology, genomics, and clinical timelines.
  • Modular agentic framework integrating domain FM-based tools as selectively callable components within LLM reasoning loops.
  • Systematic evaluation of 10+ open/closed LLMs and VLMs (gemma-3, GPT-4o, o4-mini, InternVL3, LLaMA, Qwen3, Mistral) under tool-free and tool-augmented conditions.
  • Release of benchmark data, annotation app, agent logs, and tools.

Results

  • Best multimodal baseline: InternVL3-78b, 69.1% ± 8.4% overall; outperforms GPT-4o (66.7%) by 2.5%.
  • Best longitudinal baseline: qwen3-32b, 67.0% ± 13.5% overall; llama33-70b reaches 66.0%.
  • Outcome/recurrence prediction: near chance (~50%) for all models even at scale; recurrence is the hardest subtask.
  • Digital pathology gap: up to 36.7 percentage points between best and worst model.
  • Tool augmentation: up to +9.0% task-level accuracy (multimodal track) and +11.2% (longitudinal track).
  • File access vs. accuracy: number of files retrieved is a stronger predictor of accuracy than model scale across most tasks.
  • Model size: inconsistently predictive — e.g., gemma-3-12b outperforms gemma-3-27b on digital pathology.

Limitations

  • Small patient cohorts: 26 (multimodal) and 40 (longitudinal) patients; wide confidence intervals on many tasks (±8–14%).
  • Multimodal Q&A generated via GPT-4o with only select expert review — potential annotation noise.
  • Non-persistent file access across turns is a design simplification that may not reflect all real clinical workflows.
  • Tool augmentation gains are task-specific and modest; outcome/recurrence tasks remain near chance even with tools.
  • No radiology modality despite being listed as an MTB component.
  • MSK-CHORD is CC BY-NC-ND 4.0, restricting commercial use of the longitudinal track.

Relevance to Harnesses / Meta-Harnesses

MTBBench exemplifies a domain-specific evaluation harness where the benchmark itself is embedded inside an agentic execution loop — agents do not receive inputs passively but must orchestrate file retrieval and FM-tool calls as part of answering each question. The agentic framework is a meta-harness pattern: a coordinating LLM that selectively invokes specialized sub-models (CONCH, UNI2+ABMIL, PubMed retriever, DrugBank) at runtime, analogous to tool-use harnesses in general-purpose agent research. The non-persistent memory design and evolving timeline state mirror architectural decisions seen in stateful harness pipelines. For researchers building harnesses, MTBBench provides a concrete blueprint for benchmark-coupled agent scaffolds in high-stakes domains where ground-truth evaluation requires both task correctness and tool-use strategy assessment.