Skip to content

Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks

🕒 Published (v1): 2026-07-14 02:28 UTC · Source: Arxiv · link

Why this paper was selected

Data leakage audit in WSI pathology VLM benchmarks; questions benchmark validity; critical for medical VLM evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Whole-slide image (WSI) VQA benchmarks derived from TCGA exhibit 92.3–100% case-level train–test overlap and near-complete institutional overlap, rendering reported zero-shot accuracies unreliable. The authors formalize two hierarchical leakage levels — patient-level (\(L_{\text{patient}}\)) and institutional/TSS-level (\(L_{\text{TSS}}\)) — prove both are linearly decodable from frozen encoder features, and demonstrate a 3–8 pp accuracy inflation on leaked vs. audit-clean cases within the same benchmark. Current WSI VQA evaluation cannot distinguish multimodal reasoning from memorized retrieval.

Problem

The WSI VLM ecosystem sources both training corpora and evaluation benchmarks almost exclusively from TCGA, yet no prior work has systematically audited the resulting train–test contamination at the multimodal level. Existing leakage audits are confined to unimodal CNNs and study each contamination level in isolation; none cover the instruction-tuning corpora and benchmarks used by WSI VLMs, where a contaminated case carries an entire structured VQA record rather than a single label.

Method

The audit proceeds in four stages:

  1. Overlap quantification. TCGA barcodes encode a three-level hierarchy: TSS → case → slide (Eq. 1). For each training corpus × benchmark pair the authors compute case-overlap ratio \(\rho_{\text{case}} = |C_{\text{test}} \cap C_{\text{train}}| / |C_{\text{test}}|\) and TSS-overlap ratio \(\rho_{\text{TSS}} = |T_{\text{test}} \cap T_{\text{train}}| / |T_{\text{test}}|\) (Eq. 5), yielding a two-panel contamination matrix across 8 training corpora and 4 TCGA-derived benchmarks.

  2. Feature-space probing. On 6,756 slides with features from both CONCH-v1 (mean-pooled patch encoder) and TITAN (native slide-level encoder), two experiments are run: (a) pairwise cosine similarity stratified by cross-TSS / intra-TSS / intra-patient proximity; (b) linear probing via logistic regression for patient re-identification (ReID, top-1 accuracy vs. chance) and TSS detection (one-vs-rest macro AUROC) on frozen embeddings.

  3. Cross-paper meta-analysis. Published accuracy figures from six VLMs (CVPR/ECCV/ICLR/arXiv) are tabulated alongside their audit-verified \(\rho_{\text{case}}\) on each benchmark.

  4. Within-benchmark stratification. Using the published WSI-LLaVA checkpoint, inference is run on SlideBench-VQA-TCGA and accuracy is stratified by leakage status (leaked vs. audit-clean slides identified by TCGA case IDs) across cancer type and question category to isolate the inflation effect from task-difficulty confounds.

Key Contributions

  • First joint audit of patient-level and TSS-level train–test contamination across the public WSI VLM instruction-tuning and evaluation ecosystem, with a reproducible pipeline operating on canonical TCGA identifiers.
  • Formal two-level leakage taxonomy: \(L_{\text{patient}}\) (same case in train and test) and \(L_{\text{TSS}}\) (same institution, disjoint patients), with the three-way exhaustive partition \(D_{\text{test}} = L_{\text{patient}} \cup L_{\text{TSS}} \cup D_{\text{clean}}\) (Eq. 4).
  • Empirical proof that both leakage levels are linearly encoded in the feature spaces of two structurally distinct frozen encoders (CONCH-v1 and TITAN), with TITAN exhibiting stronger leakage signal despite being a slide-native encoder.
  • Within-benchmark controlled experiment showing 3–8 pp accuracy inflation on leaked vs. audit-clean cases using the same model, benchmark, cancer types, and question categories.
  • Methodological guidelines: strict slide/patient/TSS deduplication, mandatory provenance disclosure, and use of independently curated, non-TCGA multi-center holdout sets.

Results

  • Case-level overlap: WSI-Bench-train vs. WSI-VQA-test: \(\rho_{\text{case}} = 100\%\); WSI-Bench-train vs. SlideBench-VQA-TCGA: \(\rho_{\text{case}} = 92.3\%\); PathGen-1.6M vs. SlideBench-VQA-TCGA: \(\rho_{\text{case}} = 94.3\%\); WSI-Bench internal train/test split: \(\rho_{\text{case}} = 83.1\%\).
  • TSS-level residual leakage after case-level dedup: WSI-VQA train/test achieves \(\rho_{\text{case}} = 0\%\) but \(\rho_{\text{TSS}} = 100\%\); SlideInstruction+ achieves \(\rho_{\text{case}} = 0\%\) against all four benchmarks yet retains 10.1–11.3% TSS overlap.
  • Feature-space similarity (excess over cross-TSS baseline): CONCH-v1 intra-TSS +0.183, intra-patient +0.272; TITAN intra-TSS +0.506, intra-patient +0.669; all transitions significant at \(p < 10^{-200}\) (Mann-Whitney U).
  • Linear probing: Patient ReID: CONCH-v1 14.0% top-1 (44.9× chance), TITAN 20.3% (65.0× chance); TSS detection AUROC: CONCH-v1 0.955, TITAN 0.980.
  • Accuracy inflation (within-benchmark, WSI-LLaVA on SlideBench-VQA-TCGA): Leaked vs. clean gap: +7.5 pp (Microscopy Qs), +6.3 pp (UCEC stratum), +3.8 pp (UCEC+COAD pool), +3.4 pp (Diagnosis Qs); direction positive in all four independent subsets.
  • Meta-analysis: Highest reported accuracies (MLLM-HWSI 89.6% on SlideBench, 69.2% on WSI-VQA; WSI-LLaVA 82.5%) co-occur with \(\rho_{\text{case}} \geq 92.3\%\).

Limitations

  • Upstream encoder pretraining leakage (e.g., through patch-level web corpora such as Quilt-1M and OpenPath) is acknowledged but not audited, because those corpora carry no canonical TCGA patient/slide identifiers, making \(\rho_{\text{case}}\) and \(\rho_{\text{TSS}}\) undefined.
  • The within-benchmark inflation estimate (3–8 pp) comes from a single published checkpoint (WSI-LLaVA); generalization to other VLMs is supported by meta-analysis correlation but not directly measured.
  • The audit is scoped to the instruction-tuning stage; it does not quantify leakage through the visual encoder's own pretraining corpus, which is often undisclosed or proprietary.
  • Clean audit status for models with patch-level or non-TCGA training (e.g., Quilt-LLaVA) is characterized as a lower bound, not a certificate, since identifier-based matching cannot resolve overlap in tile-level corpora.
  • Only two encoders (CONCH-v1, TITAN) are probed; it is not demonstrated whether lighter-weight or non-pathology encoders encode institutional signals to the same degree.

Relevance to Vision-Language Models

This paper directly undermines the validity of a growing body of WSI VLM benchmarks (SlideChat, WSI-LLaVA, MLLM-HWSI, PathGen-LLaVA) that have appeared at top venues, showing that the LLaVA-style offline-encoder paradigm universally used in computational pathology VLMs inherits and amplifies patient and institutional identity signals from frozen encoders. For VLM researchers, the key methodological finding is that frozen encoder features are the effective input to the language model, so any confounder embedded at the encoder stage cannot be removed by downstream instruction tuning — a structural constraint that applies broadly to any frozen-vision-tower VLM. The work motivates stricter provenance and deduplication requirements for multimodal benchmarks and suggests that "zero-shot" evaluation claims in domain-specific VLMs require explicit cross-dataset identifier auditing to be credible.