MedInsightBench: Evaluating Medical Analytics Agents Through Multi-Step Insight Discovery in Multimodal Medical Data

🕒 Published (v1): 2025-12-15 13:10 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedInsightBench is the first benchmark for evaluating multi-step medical insight discovery, comprising 332 TCGA cancer cases (3,933 insights) pairing pathology WSIs with report-derived question-insight pairs. Alongside it, MedInsightAgent is a three-module multi-agent framework that structures discovery as iterative root-question generation, image-grounded answering, and follow-up question composition. Existing LMMs score poorly (F1 G-Eval ~0.32–0.44), while MedInsightAgent substantially improves over direct LMM decoding.

Problem

Existing medical AI benchmarks probe surface-level retrieval (fact recall, direct QA) and do not measure higher-order clinical cognition: uncovering occult pathological relationships, integrating multi-modal evidence for prognostic inference, or synthesizing actionable insights from raw pathology images and reports. No dataset for this deeper "insight discovery" task existed, and standard single-query LMM evaluation cannot assess multi-step analytical depth.

Method

Benchmark construction: WSIs from TCGA are downsampled to PNG; paired PDF pathology reports are OCR-extracted and LLM-decomposed into evidence snippets → insights (six typed categories) → goal-directed questions → overarching analytical goal. LLM-assisted generation is followed by human verification at each stage.

MedInsightAgent decomposes insight discovery into three chained agents with an exploration-depth parameter p controlling iteration rounds: 1. Visual Root Finder (VRF): image-summarization module extracts visual features/keywords; web-retrieval module fetches top-10 domain literature items; root question generator synthesizes m questions from (image, goal, features, docs). 2. Analytical Insight Agent (AIA): for each question, PathGen-LLaVA (LLaVA fine-tuned on PathGen pathology data) extracts question-specific image evidence; a generator produces grounded answer + clinical insight. 3. Follow-Up Question Composer (FQC): generates n candidate follow-up questions per root question, selects the highest-scoring one, passes it back to AIA; repeats for p rounds. Total insights = r × (p + 1).

Evaluation uses Insight Recall, Precision, F1 (both ROUGE-1 and G-Eval scoring averaged over GPT-3.5-Turbo and Gemini 2.5 Pro), plus Insight Novelty (Original + Innovation scores).

Key Contributions

• MedInsightBench: 332 multi-modal cases, 3,933 human-verified insights, four difficulty levels, six insight categories, derived from TCGA cancer genomics data.
• Four-metric evaluation protocol (Recall/Precision/F1/Novelty) that captures both quality and discovery of unannotated insights, addressing bias limitations of single-LLM G-Eval.
• MedInsightAgent: domain-specific three-module multi-agent harness with iterative follow-up questioning and integrated web retrieval + specialized pathology VLM.
• Empirical analysis across 5 LMMs + 2 agent frameworks; ablation study quantifying each module's contribution.

Results

Main benchmark (G-Eval F1): - GPT-4o direct: 0.325; GPT-5: 0.332; Deepseek-VL2: 0.360; Qwen2.5-VL-32B: 0.437; InternVL3-38B: 0.367 - ReAct (GPT-4o): 0.332 — marginal gain over direct GPT-4o - MedInsightAgent (GPT-4o): 0.384 — substantial gain over GPT-4o direct - MedInsightAgent (Qwen2.5-VL-32B): 0.546 — strongest overall result

Ablation (MedInsightAgent GPT-4o, G-Eval F1 → Novelty Innovation): - Full: 0.384 / 0.270 - w/o Image-Analysis Tool: 0.353 / 0.261 — largest drop in F1 - w/o Web-Retrieval Module: 0.361 / 0.239 — largest drop in novelty - w/o Follow-Up Question Composer: 0.338 / 0.233 - w/o Image-Summarization Module: 0.378 / 0.253

Data quality (100 sampled instances): human annotation correctness 0.919, rationality 0.891, coherence 0.930; low redundancy (Distinct-2 = 0.935 for insights).

Limitations

• Benchmark sourced solely from TCGA; cancer pathology only — generalizability to other modalities (radiology, EHR) or disease types is untested.
• 332 cases is small relative to the diversity of cancer types in TCGA; potential sampling bias.
• Evaluation relies on LLM judges (GPT-3.5-Turbo, Gemini 2.5 Pro) for G-Eval scoring, introducing model-specific bias even when averaged.
• MedInsightAgent requires PathGen-LLaVA as a hard dependency; performance is coupled to this specialized VLM's coverage of pathology.
• Precision consistently exceeds recall across all methods, indicating agents produce redundant high-confidence insights rather than broad exploration; depth coverage remains limited.
• Follow-up question diversity is not explicitly controlled — iterative rounds may converge on similar angles.

Relevance to Harnesses / Meta-Harnesses

MedInsightAgent is a concrete domain-specific multi-agent harness that operationalizes a reusable orchestration pattern: gather context → generate task decomposition → execute subtasks with specialized tools → iteratively deepen via follow-up. The exploration-depth parameter p makes the harness configurable in the same spirit as meta-harness depth controls. The ablation study directly quantifies each harness module's contribution to end-task quality, providing empirical evidence for the value of modular harness design over monolithic LMM prompting. For researchers tracking harnesses, this paper demonstrates how domain specialization (web retrieval + pathology VLM) can be wired into a generic iterative-questioning scaffold, and that the harness structure itself (not just the backbone LMM) is responsible for the bulk of the quality gain over ReAct-style baselines.