SVBench: A Benchmark with Temporal Multi-Turn Dialogues for Streaming Video Understanding

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SVBench is a benchmark for evaluating large vision-language models (LVLMs) on long-context streaming video understanding via temporal multi-turn QA chains. It introduces 49,979 QA pairs across 1,353 streaming videos with explicit temporal linkages between dialogue chains. Experiments on 14 models reveal severe gaps in streaming video comprehension even for state-of-the-art models.

Problem

Existing video QA benchmarks rely on single-turn, isolated QA pairs or short clips and cannot evaluate whether LVLMs can sustain coherent temporal reasoning across the full duration of a streaming video. No prior benchmark captures the continuous, dynamic nature of live video with multi-turn dialogues that reference prior context across clip boundaries.

Method

SVBench uses a semi-automated annotation pipeline: (1) raw videos from 6 platforms are filtered for quality and scene diversity, then split into clips via PySceneDetect; (2) GPT-4o generates QA chains of 5–6 turns per clip, followed by human refinement; (3) GPT-4 scores each chain across 7 quality dimensions with a 90-point threshold before acceptance; (4) LLM-identified and human-verified temporal linkages connect adjacent QA chains into temporal dialogue paths. Two evaluation protocols are defined — dialogue evaluation (all prior QA context provided) and streaming evaluation (80% probability of cross-chain jumps via temporal linkages, simulating live interaction). A baseline model, StreamingChat (InternVL2 + InternViT + InternLM2 + LoRA fine-tuned on SVBench training data), is also released.

Key Contributions

• First benchmark specifically targeting streaming video understanding with temporal multi-turn QA chains (49,979 QA pairs, 1,353 videos, avg. 36.94 QA pairs/video — highest among known video datasets).
• Novel task formulation: QA chains per clip connected by explicit temporal linkages, enabling cross-clip temporal reasoning evaluation.
• Two evaluation protocols (dialogue and streaming) assessed across five metrics: Semantic Accuracy (SA), Contextual Coherence (CC), Logical Consistency (LC), Temporal Understanding (TU), Informational Completeness (IC).
• Nine skill categories for fine-grained analysis: Intention Inference, Potentiality Assessment, Counterfactual Reasoning, Spatio-Temporal Speculation, Relationship Inference, Character State and Transition, Comparison and Trend Analysis, Common Sense Inference, Event-Centric Analysis.
• StreamingChat baseline showing that fine-tuning on temporal dialogue paths yields significant streaming gains without degrading general VL benchmarks.

Results

• GPT-4o achieves the highest scores: OS 66.29 (dialogue), 58.17 (streaming); GPT-4V is comparable (65.19 / 57.35).
• Among open-source models, StreamingChat leads: OS 59.41 (dialogue) / 53.90 (streaming), a +28.79% / +26.20% improvement over the base InternVL2 model it is built on.
• MiniCPM-V 2.6 is the second-best open-source model: OS 54.30 / 52.19.
• Streaming evaluation scores are consistently 5–10 points below dialogue evaluation across all models, indicating cross-chain temporal reasoning is harder.
• Nearly all models score below 60 across all 9 skill categories; Spatio-Temporal Speculation (STS) and Counterfactual Reasoning (CR) are the weakest skills even for GPT-4o (44.58 and 49.97 respectively).
• StreamingChat surpasses GPT-4V and GPT-4o in Potentiality Assessment (71.96 vs. 61.57 / 59.47) and Event-Centric Analysis.
• Ablation: multi-turn QA context consistently boosts METEOR and GPT4-Score over single-instance evaluation for all 14 models tested.

Limitations

• StreamingChat is built on InternVL2 and shows slight regressions on MMBench (image) and MMBench-Video versus the base model after fine-tuning.
• The streaming evaluation uses an 80% probabilistic jump rule to simulate temporal linkages, which is a proxy for real streaming interaction rather than a live inference setting.
• QA chain generation relies on GPT-4o, introducing potential biases from the generator model into the benchmark itself.
• Videos are capped at those with 5–15 scenes, excluding very complex or very simple content, limiting generalizability.
• StreamingChat processes video at only 1 FPS within a 32k context window; very long or fast-changing streams may not be fully captured.

Relevance to Vision-Language Models

SVBench directly probes an under-evaluated capability of LVLMs: maintaining coherent temporal reasoning over continuous video streams with multi-turn dialogue history, which is crucial for real-world deployment in live-stream monitoring, assistive agents, and interactive video systems. The benchmark exposes that current top-tier open-source LVLMs lag substantially behind GPT-4o on streaming tasks, providing a concrete target for video-LLM development. The StreamingChat results demonstrate that fine-tuning with temporal dialogue path data (even from the same architecture) yields large gains, suggesting data format is as important as model scale. The skill taxonomy (9 categories) offers a diagnostic framework aligned with emerging interest in compositional and temporal reasoning within the VLM community.