V2P-Bench: Evaluating Video-Language Understanding with Visual Prompts for Better Human-Model Interaction¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; visual prompt-based video benchmark enables richer human-model interaction

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

V2P-Bench is a benchmark of 980 videos and 1172 QA pairs designed to evaluate Large Vision-Language Models (LVLMs) on video understanding when queries are specified via visual prompts (bounding boxes, arrows, masks, etc.) rather than text. Evaluation of 15 models reveals that even o1 scores only 71.8% against a human expert ceiling of 88.3%, and that models systematically "hack" answers—especially on long videos with sparse frame sampling.

Problem¶

Existing video benchmarks rely exclusively on text prompts, which require users to encode spatial/temporal references in natural language and force models to decode ambiguous referential descriptions before answering. This double-encoding bottleneck is particularly severe in complex multi-object scenes and does not reflect intuitive human–model interaction. No prior benchmark comprehensively evaluates LVLMs on video inputs where the query target is indicated directly on a frame via visual overlays.

Method¶

V2P-Bench collects 980 videos from twelve existing datasets, reclassified into twenty categories spanning short (seconds) to long (up to 2 hours) durations. Human annotators write multiple-choice QA pairs and place exactly one visual prompt annotation per question on a single frame; visual prompt types include rectangle, mask contour, ellipse, triangle, scribble, point, arrow, and Set-of-Mark (SoM). Questions cover three task tiers—Basic Perception (object/human attributes), Temporal Understanding (direction, feature mapping, forward/reverse temporal, action sequence, spatial relationship, counting), and High-level Reasoning (causal, plot, counterfactual)—yielding twelve evaluation dimensions. Dataset quality is enforced via (1) blind LLM filtering (GPT-4o and Gemini-2.5-Pro answering text-only QA pairs at temperature 0.2 across two rounds; items answered correctly in both rounds are removed) and (2) manual/rule-based review with option-order shuffling to produce balanced answer distributions (≈25% each option).

Key Contributions¶

First video benchmark to pair QA with single-frame visual prompts for realistic human–model interaction evaluation, covering 8 visual prompt types and durations from 3 s to 2 h.
Empirical demonstration that visual prompts outperform text prompts for both models (up to +15.1% for Gemini-2.5-Pro) and users (task time −7.7 s, satisfaction +2.4 points, 64.5% user preference for visual prompts).
Systematic characterization of hack phenomena: models select options without genuine video comprehension; hack rate increases with video length (e.g., Qwen2.5-VL-7B: 11.1% → 23.0% → higher for short/medium/long) and decreases with denser frame sampling.
Comprehensive evaluation of 15 LVLMs (3 closed-source, 12 open-source), revealing a large gap between the best model (o1, 71.8%) and human experts (88.3%), with most open-source models below 60%.

Results¶

Human expert accuracy: 88.3%
Best model (o1): 71.8% overall; GPT-4o: 65.4%; Gemini-2.5-Pro: 69.8%
Best open-source: Qwen2.5-VL-72B at 59.8%; InternVL3-8B at 61.7%
Most open-source 7B models: 46–55% range
Blind answering (no video): GPT-4o 1.4%, Gemini-2.5-Pro 9.6%, Qwen2.5-VL-7B 3.0%—confirming effective filtering
Weakest dimensions (all models): Object Direction (avg 34.4%) and Spatial Relationship (avg 45.7%)
Visual vs. text prompt comparison: Converting visual prompts to text drops accuracy by 9–15 percentage points across all tested models
User study (N=20, 10 Q each format): Accuracy 69.5% (visual) vs. 57.0% (text); completion time 18.1 s vs. 25.2 s; satisfaction 7.5 vs. 5.3
Hack rate (shuffled video–question pairs): only 6.4% (Qwen2.5-VL-7B) and 3.9% (MiMo-VL-7B) trigger refusal; remaining cases produce guessed answers

Limitations¶

Visual prompt annotation covers only one frame per QA pair by design, so the benchmark cannot test models' ability to track or ground visual prompts across frames.
All videos are repurposed from existing benchmarks, potentially inheriting their biases in scene type and subject matter.
The hack-phenomena analysis is limited to two open-source models (Qwen2.5-VL-7B, MiMo-VL-7B); closed-source model hacking behavior is not fully characterized.
LLaVA-NeXT-INST-IT fine-tuned on SoM-only data shows catastrophic forgetting on other visual prompt types, but no remedy or training recipe is proposed.
Frame sampling rates vary across models (4–128 frames), making direct comparisons across model classes imprecise.

Relevance to Vision-Language Models¶

V2P-Bench directly probes a practical gap in VLM evaluation: whether models can localize and reason about visually-indicated referents in video, a modality that prior benchmarks conflate with text-based spatial description. The benchmark's finding that spatiotemporal understanding (object direction, spatial relationships) remains the hardest dimension even for frontier models identifies a concrete sub-capability deficit orthogonal to general video QA. The hack-phenomena analysis is broadly relevant to the VLM benchmarking community, showing that low frame-sampling density systematically inflates accuracy scores and that models are poorly calibrated to abstain. For researchers developing visual-prompt-aware video models or studying grounded video understanding, V2P-Bench's eight-type visual prompt taxonomy and instance-level, multi-duration design offer a more realistic testbed than existing text-only suites.