Skip to content

MedStreamBench: A Time-Aware Benchmark for Streaming and Proactive Medical Video Understanding

🕒 Published (v1): 2026-07-02 06:07 UTC · Source: Arxiv · link

Why this paper was selected

Time-aware medical video benchmark; proactive prediction timing critical for clinical AI systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MedStreamBench is a time-aware benchmark for medical video understanding that evaluates whether models answer at the right time, not just correctly. It integrates 22 medical datasets into 5,419 QA instances across four temporal modes—retrospective, present, future, and proactive—with strict evidence-window constraints. Experiments show a substantial drop in model performance when moving from offline to streaming/proactive settings.

Problem

Existing medical video benchmarks assume full-video access and measure only semantic correctness of final answers, ignoring temporal discipline. Real clinical AI systems must also decide when to answer, defer, or proactively raise alerts—a requirement that offline evaluation frameworks cannot capture. No prior benchmark covers prefix-only streaming answering or proactive alert triggering for medical video.

Method

Each benchmark item follows the schema \(x = (v, q, a, \tau, m, r, s)\), where \(\tau\) encodes query time \(t_q\), context window \(w\), and input range \([t_s, t_e]\). Four temporal modes are defined: - Retrospective/Present (single-turn): model receives \(I = [\max(0, t_q - w),\ t_q]\) and must answer or return unanswerable. - Future (single-turn/streaming): same input window but the target event occurs after \(t_q\); streaming rounds extend as \(I_k = [t_q, t_c^{(k)}]\). - Proactive (streaming): per-round prefixes \([t_q, t_c^{(k)}]\) with response space \(\{\)no alert, uncertain, alert: <reason>\(\}\); alerts triggered only when sufficient visual evidence appears.

Three metrics are reported: content correctness \(C\) (normalized exact match or rubric-based judge scoring correctness + evidence grounding + safety ∈ [0,1]); responsiveness \(R\) (penalizes early, delayed, or missed first positive response relative to expected time \(t^*\)); and stability \(S = \frac{1}{|K_{\geq t^*}|}\sum_{k \in K_{\geq t^*}} \mathbf{1}[y_k \text{ is positive}]\). The overall streaming score is \(O = 0.7C + 0.3R\).

A three-stage data engine harmonizes heterogeneous source annotations (structured labels, directory labels, weak mining) into unified JSONL, validates temporal contracts, and expands streaming items into per-round inference jobs (1 frame per 2 s).

Key Contributions

  • First benchmark to evaluate streaming medical video understanding with per-round prefix-only evidence windows.
  • First proactive medical video QA paradigm requiring models to determine whether and when to trigger clinical alerts.
  • Unified temporal QA schema spanning 22 source datasets, 5,419 QA records (4,663 single-turn + 756 streaming → 3,369 round-level jobs) across surgical, endoscopic, and skill-assessment domains.
  • Three-metric evaluation separating semantic correctness, temporal responsiveness, and post-evidence stability.
  • Reproducible data construction and inference engines with standardized frame sampling and prompt protocols.

Results

  • Gemini-2.5-Pro is the strongest overall (score 0.4742); InternVL3.5-8B leads among open-source (0.2713); Lingshu-7B leads among medical-domain models (0.2586).
  • All models degrade substantially in streaming/proactive settings: e.g., Gemini-2.5-Pro future content = 0.4384 but proactive content = 0.4494; open-source proactive content scores fall to 0.05–0.23.
  • Responsiveness scores are highly variable: LLaVA-OneVision-7B achieves 0.8142 future responsiveness but only 0.0579 proactive content, indicating spurious early alerting rather than genuine detection.
  • Organ-level analysis (Figure 4) shows no single model dominates across all clinical domains; InternVL3.5-8B leads on GI tract/capsule, Qwen2.5-VL-7B on multi-organ open surgery, HuluMed-4B on robotic surgery.
  • A preliminary human spot-check confirmed alignment between AI-generated answers and human annotators (exact figures truncated in provided text).

Limitations

  • Source datasets vary substantially in annotation granularity, label quality, and temporal density; some items derive from directory labels, hybrid generation, or weak supervision rather than expert-authored QA.
  • Not all weakly supervised or model-assisted samples received exhaustive manual visual review.
  • Fixed 1-frame-per-2-second sampling protocol may underrepresent brief transient events.
  • Multi-source composition inherits distributional biases in procedure type, device, and recording style; aggregate scores should not substitute for domain-level analysis.
  • Benchmark is research-only and explicitly not a clinical decision-support system.

Relevance to Vision-Language Models

MedStreamBench directly stresses VLMs on a capability—temporally grounded decision-making under evidence constraints—that is absent from standard offline video-QA benchmarks like Video-MME or MVBench. The revealed gap between retrospective recognition scores and streaming/proactive scores quantifies how poorly current VLMs (including Gemini-2.5-Pro and domain-fine-tuned medical models) generalize from full-context inference to prefix-only, real-time scenarios. For researchers working on medical or video VLMs, the benchmark also introduces a practically relevant proactive monitoring paradigm and multi-dimensional metrics (correctness, responsiveness, stability) that could inform training objectives beyond next-token prediction accuracy. It complements concurrent work such as SurgOnAir and MedHorizon by being the first to cover all four temporal modes simultaneously with strict evidence-window enforcement.