Skip to content

PathAgentBench: Benchmarking Evidence-Seeking Vision-Language Models on Whole-Slide Pathology Image

🕒 Published (v1): 2026-07-21 16:33 UTC · Source: Arxiv · link

Why this paper was selected

PathAgentBench; evidence-seeking VLMs evaluated on multi-scale whole-slide pathology

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PathAgentBench benchmarks vision-language models (VLMs) on four evidence-seeking stages of whole-slide image (WSI) diagnosis—interpretation, verification, acquisition, and integration—organized as a hierarchical diagnostic tree over 1,822 TCGA WSIs with 17,135 pathologist-annotated paths. It reveals a sharp capability asymmetry: leading models exceed 93% accuracy on multi-scale reasoning over pre-supplied evidence yet achieve mIoU below 0.09 when required to localize diagnostic regions in gigapixel slides.

Problem

Existing pathology benchmarks—whether patch-level (PathVQA, PathMMU) or slide-level (WSI-Bench, SlideBench)—evaluate models only after relevant evidence has been selected or pre-extracted, leaving the evidence-acquisition step (deciding where to look in a gigapixel slide) unmeasured. Strong VQA performance therefore does not establish whether a model can perform end-to-end diagnostic exploration, and failure modes within the diagnostic workflow cannot be isolated.

Method

WSI diagnosis is formalized as a diagnostic tree \(T_s = (V_s, E_s)\) over a three-level image pyramid (\(2.5\times \to 10\times \to 40\times\)). Each node \(v = (s, \ell, b)\) pairs a magnification \(\ell\) with a canvas bounding box \(b\); root-to-leaf paths \(\pi\) accumulate per-node findings \(d(v)\) into a path-level diagnosis \(D(\pi)\). The joint model is:

\[p_\theta(y \mid s) = \sum_{\pi \in \mathcal{P}(T_s)} p_\theta(y \mid \Pi=\pi, s)\, p_\theta(\Pi=\pi \mid s)\]

separating the evidence-reasoning term from the evidence-acquisition policy.

Four tasks operationalize this decomposition:

  • T1 – Evidence Interpretation (image-to-text matching): given a diagnostic patch, select the correct pathologist-authored finding from four same-organ, same-magnification distractors mined by greedy cosine-dissimilarity selection with all-MiniLM-L6-v2 (similarity threshold 0.6). Yields 10,284 four-way MCQs.
  • T2 – Evidence Verification (text-to-image retrieval): converse direction—given a finding, select the matching patch from four same-scale candidates using identical distractor mining. Yields 10,284 four-way MCQs.
  • T3 – Evidence Acquisition (diagnostic region localization), two modes:
  • Mode A (text-guided): a LangGraph agent with tools get_image_info, extract_roi, and finish_and_report navigates under a budget of (10, 7, 5) ROI calls at (\(2.5\times\), \(10\times\), \(40\times\)); predicted box position is scored by mIoU against pathologist annotation (extent fixed to ground-truth). Evaluated on a 50-slide stratified subset (150 queries/model).
  • Mode B (autonomous exploration): agent scores all \(2.5\times\) tiles for tumor suspicion, recursively retains top-\(K\) children \((K = 6, 3, 2)\) per level; scored by conditional hit rate (recall among positive tiles reachable from retained branches) and unconditional hit rate (recall over all positive tiles at that magnification). Evaluated on 190 private breast cancer WSIs.
  • T4 – Evidence Integration (multi-scale reasoning): given organ label and three per-magnification findings along a path, select the correct integrated diagnosis from four options under Diagnose, Triage, or Refine sub-tasks. Yields 51,167 four-way MCQs.

Annotations were produced by ten board-certified pathologists using a dual-review workflow; only slides passing supervisor review are released.

Key Contributions

  • Formalizes WSI diagnosis as a hierarchical diagnostic tree that explicitly decouples evidence acquisition (\(p_\theta(\Pi \mid s)\)) from evidence reasoning (\(p_\theta(y \mid \Pi, s)\)).
  • Introduces PathAgentBench with four complementary tasks spanning 1,822 TCGA WSIs across 16 organ types and 17,135 pathologist-annotated paths, plus a 190-WSI private breast cohort for agentic exploration.
  • Provides the first benchmark jointly measuring interpretation, verification, spatial acquisition, and multi-scale integration under a common diagnostic-tree framework.
  • Evaluates 20 models (general-purpose, general-medical, pathology-specialised) and quantifies where current pathology agents break down.

Results

  • T4 (multi-scale reasoning): Gemini-3-Flash 91.18%, GPT-5.2 89.20%; human reference 96.6%. Leading open-weight models exceed 93% in this pre-supplied-evidence setting.
  • T1 (image-to-text): Gemini-3-Flash 63.48%, GPT-5.2 52.23%; both exceed 50% overall.
  • T2 (text-to-image): Gemini-3-Flash 67.74%, GPT-5.2 60.96%; all leading models exceed 50%.
  • T3-A (text-guided localization): Best VLM mIoU at \(2.5\times / 10\times / 40\times\): GPT-5.2 0.086 / 0.056 / 0.066; Gemini-3-Flash 0.038 / 0.089 / 0.068. The non-VLM parent-box center heuristic achieves 0.247 / 0.283 at \(10\times\) / \(40\times\), outperforming all tested VLMs.
  • T3-B (autonomous exploration): Unconditional hit rate collapses from 0.522 at \(2.5\times\) to 0.185 at \(10\times\) and 0.020 at \(40\times\) for the best-performing model. Conditional hit rates are higher but still degrade substantially across magnifications.
  • Evidence acquisition is identified as the principal bottleneck; all evidence-reasoning tasks (T1, T2, T4) substantially outperform acquisition (T3).

Limitations

  • Mode A (text-guided localization) is evaluated on only 50 slides (150 queries per model) due to the per-query cost of gigapixel tool-call trajectories.
  • Mode B is restricted to a single disease site (breast cancer), limiting generalizability of autonomous-exploration results across organ types.
  • Only four models (three VLMs + Patho-R1-7B) produced valid bounding-box tool calls for Mode A; the majority of open-weight models could not be evaluated on this task.
  • The private Mode B cohort has no multi-scale path structure, restricting it to exploration evaluation rather than full diagnostic-tree assessment.
  • Results table was truncated in the provided text; complete open-weight and medical-model comparisons are not fully visible.

Relevance to Vision-Language Models

PathAgentBench directly exposes a fundamental gap in VLM capability: spatial grounding and active evidence acquisition in high-resolution medical images remain far below the level of reasoning over pre-supplied evidence, with the best models scoring mIoU below 0.09 against a center-heuristic baseline that scores 0.283. This benchmark is immediately actionable for researchers developing or evaluating pathology VLMs, agentic systems that combine VLMs with tool use and hierarchical navigation, or any multimodal model whose deployment requires self-directed region selection rather than passive patch interpretation. The diagnostic-tree formalism and the separation of \(p_\theta(\Pi \mid s)\) from \(p_\theta(y \mid \Pi, s)\) also provide a principled evaluation decomposition transferable to other domains where active spatial search precedes reasoning.