VIBE: Annotation-Free Video-to-Text Information Bottleneck Evaluation for TL;DR¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VIBE is an annotation-free framework for scoring and selecting VLM-generated video summaries using information-theoretic metrics derived from the information bottleneck principle. It ranks candidate summaries by grounding (how well a summary aligns with visual content) and utility (how informative the summary is for a downstream task), without requiring human-annotated reference captions. In human studies with 243 participants across three datasets, VIBE-selected summaries improve task accuracy by up to 61.23% and reduce response time by up to 75.77% versus naive VLM outputs.

Problem¶

Existing video caption evaluation metrics (ROUGE, BLEU, CIDEr) require costly human-annotated reference captions and are task-agnostic—they measure lexical similarity to a gold standard rather than whether the summary actually helps a human complete a downstream task. This creates a circular problem: annotation is expensive precisely because humans must watch the videos, which summarization is meant to avoid. Additionally, VLMs frequently produce verbose, redundant captions that inflate cognitive load without improving decision-making.

Method¶

VIBE instantiates the information bottleneck (IB) objective as two pointwise mutual information (PMI) scores computed via masked inference over a VLM's next-token prediction probabilities—no retraining required.

Grounding score approximates I(V; T): given a summary T with tf-idf keywords replaced by <MASK> tokens (Tmasked), the score is:

log P(T | V, Tmasked) / P(T | Tmasked) Two VLM inferences are run—one with and one without the video—and the log-ratio measures how much the video helps reconstruct the masked summary beyond what the text alone provides.

Utility score approximates I(T; Y): given video frames with key regions randomly cropped out (Vmasked), the score is:

log P(Y | Vmasked, T) / P(Y | Vmasked) This measures how much the summary T compensates for the missing visual information when predicting the task answer Y. Task labels Y are needed but not human-annotated summary labels.

Selection: k candidate summaries (k=5, sampled at varied temperatures) are ranked by a convex combination α·IP(V;T) + β·IP(T;Y). Sweeping α ∈ {0, 0.05, …, 1} with β = 1−α traces a Pareto front; VIBE-selected summaries lie on this frontier while naive VLM outputs fall strictly inside it. The backbone used throughout is Qwen2.5-VL-72B-AWQ.

Key Contributions¶

Defines the annotation-free, task-aware video summary evaluation problem, framing it as an IB selection problem over VLM outputs.
Proposes two PMI-based scores (grounding, utility) computable via masked inference on any VLM with next-token probability access—closed- and open-source compatible.
Introduces LearningPaper24, a curated dataset of 2,287 video presentations from ICLR 2024 and NeurIPS 2024 with associated primary-area classification tasks.
Validates VIBE through a between-subjects human study (n=243) measuring accuracy, response time, and inverse efficiency score (IES = RT/accuracy).

Results¶

LearningPaper24 (250–325 s videos): Max-U accuracy 37.89% vs. Naive 28.42% (+9.47 pp); response time 46.69 s vs. Video Only 192.73 s (−75.77%); IES 1.28 vs. 9.85 for Video Only.
LongVideoBench (221–489 s): Max-U accuracy 65.00% vs. Naive 46.43% (+18.57 pp, +40%); Max-G 59.23% (+12.8 pp, +27.6%); both significantly better than Video Only (74.44% accuracy but IES 2.93 vs. Max-U 1.14).
SUTD-TrafficQA (2–10 s): Max-U accuracy 84.81% vs. Naive 76.96% (+7.85 pp; t(25)=3.311, p=0.001, d=1.325); response time gains minimal given short clip duration.
Utility score positively correlates with human accuracy across all datasets (Spearman rs = 0.684/0.399/0.463, all p<0.01).
Word count strongly negatively correlates with response time per word (rs = −0.761 / −0.684 / −0.658), indicating longer summaries reduce per-word engagement.
Ablation over VLM backbone size shows consistent IB score trends, suggesting lightweight models can verify outputs of larger ones.

Limitations¶

Utility score computation requires task labels Y, limiting applicability to settings with predefined structured tasks; grounding-only mode degrades accuracy gains.
Multiple VLM inferences are required per candidate summary; inference cost scales with the number of masked tokens and the number of sampled candidates k.
PMI estimates are biased by (a) the tf-idf masking hyperparameters (threshold, keyword count) and (b) the VLM's own training distribution and probability calibration.
Optimal masking strategies for text and video remain open questions; the paper uses tf-idf for text and random 1/16 cropping for video frames, which are heuristics.
Benefits are substantially reduced for short videos (≤10 s), where watching raw video is nearly as fast as reading a summary.
Not evaluated on tasks requiring fine-grained visual detail where text summaries are inherently lossy (e.g., action recognition with subtle motion cues).

Relevance to Vision-Language Models¶

VIBE directly addresses a key weakness of VLMs as summarization agents: their tendency to generate verbose, low-utility outputs that are hard to evaluate without human annotation. By leveraging a VLM's own next-token probabilities as an evaluation signal, VIBE provides a self-referential quality metric that requires no external oracle—relevant for VLM output quality control pipelines. The annotation-free framing is particularly important for the VLM evaluation community, which currently relies on reference-based metrics (ROUGE, CIDEr) that cannot capture task utility. VIBE also demonstrates that rejection sampling over diverse VLM samples—rather than single-pass generation—is a practical inference-time strategy for improving output quality, connecting to broader work on best-of-N sampling and test-time compute scaling in multimodal models.