Skip to content

CaReBench: A Fine-grained Benchmark for Video Captioning and Retrieval

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Fine-grained video captioning and retrieval benchmark reveals VLM temporal limits

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CaReBench is a 1,000-video benchmark with human-annotated detailed captions (~228 words average) that separately annotates spatial (object) and temporal (action) content, enabling fine-grained spatiotemporal evaluation of video-language models. Two new metrics—ReBias (retrieval spatiotemporal bias) and CapST (captioning F1 over objects and events)—reveal systematic biases in current VLMs. A unified baseline (CaRE) trained via two-stage SFT on Qwen2-VL achieves competitive performance on both retrieval and captioning simultaneously.

Problem

Existing video captioning and retrieval benchmarks use short, coarse-grained captions (MSR-VTT: 9.4 words avg.; DREAM-1K: 59.3 words) insufficient to stress-test modern VLMs. Auto-annotated benchmarks (VDC via GPT-4o) introduce hallucinations and biases. No prior benchmark provides manually separated spatial/temporal captions, so the spatial vs. temporal biases of VLMs are invisible to current evaluation protocols. Additionally, captioning metrics—n-gram (CIDEr) and LLM-based (AutoDQ, VDCScore)—either fail on long captions or ignore one dimension (objects or events).

Method

Benchmark construction: 1,000 videos are selected from FineAction (106 subcategories). A two-stage annotation pipeline produces hierarchical captions of 150–300 words, covering a general overview, object description, action description, and misc. description. In Stage-II, each caption is manually separated into a pure spatial caption (objects, no motion language) and a pure temporal caption (actions, no static references).

Metrics: - ReBias: measures retrieval spatiotemporal bias as \(B = 1 - \bar{R}_\text{temporal}/\bar{R}_\text{spatial}\); lower is better (0 = balanced). - CapST: LLM extracts objects from spatial GT and events from temporal GT, then computes precision/recall via Natural Language Inference (NLI) entailment between prediction and ground truth, reporting F1. Multi-attribute descriptions are split into atomic facts before NLI to avoid partial-match penalization.

Baseline (CaRE): Built on Qwen2-VL 7B with two-stage SFT. Stage-I fine-tunes on Tarsier Recap + LLaVA-Video-178k to align the model with fine-grained text space (vocabulary space \(\mathbb{R}^{D_v}\)). Stage-II freezes the vision encoder and trains the LLM via text-only contrastive learning (NLI triplets with EOL prompt: "Summary of above sentence in one word:") using loss: $\(\mathcal{L} = -\log \frac{e^{\cos(f_i, f_i^+)/\tau}}{\sum_{j=1}^{N} e^{\cos(f_i, f_j^+)/\tau} + e^{\cos(f_i, f_i^-)/\tau}}\)$ shifting outputs to embedding space \(\mathbb{R}^{D_e}\) for retrieval.

Key Contributions

  • CaReBench: 1,000-video benchmark with fully human-annotated, hierarchically structured captions (avg. 228 words) and manually separated spatial/temporal sub-captions.
  • ReBias metric: quantifies a model's retrieval bias toward static vs. dynamic content.
  • CapST metric: joint F1 evaluation of captioning quality over both objects and events via NLI, with atomic attribute splitting.
  • CaRE baseline: unified Qwen2-VL 7B model handling both fine-grained retrieval and detailed captioning via two-stage SFT; achieves SOTA on CaReBench in both tasks at 7B scale.

Results

Video Captioning (Events F1, Overall): - CaRE: 35.1 vs. Qwen2-VL 7B: 28.8, Qwen2.5-VL 7B: 31.1, GPT-4o mini: 36.8 - CaRE (Stage-I only): 35.3; CARE (Stage-II): 35.1

Video Captioning (Objects F1, Overall): - CaRE: 31.7 vs. Qwen2-VL 7B: 24.0, Qwen2.5-VL 7B: 33.5, GPT-4o mini: 33.8

Video Retrieval (General, Text-to-Video R@1): - CaRE: 77.0 vs. best CLIP-based (InternVideo2-S2 1B): 72.5; best contrastively-trained MLLM (Qwen2.5-VL 7B): 77.5 - CaRE Video-to-Text R@1: 79.0 vs. Qwen2.5-VL contrastive: 75.5

Key bias finding: CLIP-based models show large ReBias (strong spatial, weak temporal recall); MLLMs contrastively trained on NLI exhibit near-zero ReBias but lower absolute recall than CaRE.

Limitations

  • Benchmark covers only short videos (5–20 s, from FineAction); generalization to long-form video understanding is untested.
  • 1,000 samples is relatively small; subcategory diversity is bounded by FineAction's 106 categories.
  • CaRE Stage-II uses text-only NLI data for contrastive training, introducing a modality gap that may limit retrieval of visually subtle distinctions.
  • CapST relies on an LLM for element extraction and NLI scoring, inheriting potential errors from that model.
  • Captions are English-only; cross-lingual evaluation is absent.

Relevance to Vision-Language Models

CaReBench directly targets a known weak point of video-VLMs: the inability to distinguish spatially similar videos that differ only in temporal dynamics, or temporally similar videos that differ in static composition—a gap invisible to short-caption benchmarks. The ReBias metric operationalizes the spatial/temporal imbalance that is frequently discussed but rarely measured in VLM evaluation work. The CaRE two-stage SFT approach is a concrete recipe for endowing generative MLLMs (decoder-only architectures) with discriminative embedding capability without separate retrieval heads, which is directly relevant to unifying generative and retrieval-augmented VLM paradigms. For researchers tracking VLMs, this work provides both a diagnostic benchmark and a method for closing the MLLM/CLIP retrieval gap.