Skip to content

VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Bhiksha Raj (CMU); ICLR 2026; bootstrapping scalable MLLM-as-judge for video

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoJudge is a bootstrapped framework for training compact (3B/7B) MLLM judges to evaluate video understanding model outputs. A generator–evaluator pipeline iteratively synthesizes rated response candidates, which are used to fine-tune Qwen2.5-VL into specialized judge models. VideoJudge-7B matches or outperforms Qwen2.5-VL-32B/72B on 3 of 4 meta-evaluation benchmarks, while video input—not chain-of-thought—proves critical for reliable judging.

Problem

Standard reference-based metrics (BLEU, ROUGE, BERTScore) fail to capture semantic fidelity and contextual grounding in open-ended video understanding outputs. Human evaluation is costly and inconsistent. Extending LLM/MLLM-as-a-Judge to video remains underexplored due to: (1) absence of large-scale datasets with human preference signals for video, (2) reliance on proprietary black-box models (GPT-4o), and (3) lack of principled, scalable evaluation criteria across diverse video tasks.

Method

The framework operates in two stages:

Stage 1 — Bootstrapping. Seed data (25K triplets \((v, x, y^*)\) from VideoInstruct-100K, VCG-Plus-112K, and VideoChat2-IT) is expanded into a rated dataset. Dense video descriptions \(\tilde{v}\) are generated by a strong VLM to serve as semantic proxies, reducing repeated raw-video inference. For each \((x, \tilde{v}, y^*)\), a generator \(G\) produces \(N-1 = 4\) candidate responses, each targeting a rating \(r \in \{1,\ldots,4\}\): $\(y_0^{(r)} = G(p_\text{gen} \| \tilde{v} \| x \| y^*, r)\)$ An evaluator \(E\) assigns rating \(\hat{r}\) and feedback \(f\): $\(\hat{r}, f_t^{(r)} = E(p_\text{eval} \| \tilde{v} \| x \| y^* \| y_t^{(r)})\)$ Candidates with deviation \(\Delta_t^{(r)} = |r - \hat{r}| \leq \alpha\) are accepted; others are refined using evaluator feedback up to \(T\) iterations. The gold response \(y^*\) is assigned rating 5. This yields 103,825 pointwise examples across 20,765 unique video–instruction pairs.

Stage 2 — Fine-tuning. Qwen2.5-VL (3B, 7B) is fully fine-tuned in BF16 on the bootstrapped data (NLL loss, 128K max sequence length, 1 fps, max 60 frames training / 180 evaluation). Pointwise models produce <reasoning> then <score>; an optional rubric-augmented variant (VideoJudgeR) first emits a <rubric> grounded in instance-specific criteria. Pairwise models are trained on response pairs derived from rating comparisons with randomized ordering to mitigate positional bias.

Key Contributions

  • First bootstrapped framework for training scalable MLLM-based video understanding evaluators without costly human annotation.
  • Judge models that generate instance-specific evaluation rubrics at inference time, enabling interpretable and grounded scoring.
  • Demonstration that fine-tuned small models (3B/7B) match or surpass ∼10× larger general-purpose MLLMs on video judging tasks.
  • New pointwise and pairwise meta-evaluation benchmarks (VideoJudgeLLaVA-MetaEval, VideoJudgeVCG-MetaEval, VideoJudge-Pairwise, VideoJudge-Pairwise-H) plus open-sourced models and datasets.

Results

  • VideoJudge-7B outperforms or matches Qwen2.5-VL-32B and -72B on 3/4 meta-evaluation benchmarks (VideoJudgeLLaVA, VideoJudgeVCG, LongVideoBench) for pointwise correlation (S/P up to 0.82/0.80 on VideoJudgeLLaVA vs. 0.80/0.81 for 72B).
  • VideoJudgeR-3B (rubric-augmented, trained on 10% of data): MAE 0.59, RMSE 1.05, Pearson/Spearman ~74, versus Qwen2.5-VL-3B baseline (MAE 1.15, RMSE 1.56, correlations ~38); comparable to 32B/72B (MAE 0.59/0.54).
  • LLM-only judges (Qwen3, including with chain-of-thought "thinking mode") consistently underperform MLLM judges (Qwen2.5-VL) on video benchmarks, confirming video input is essential.
  • Human evaluation of pairwise data quality: annotator agreement 94.8% (Cohen's \(\kappa = 89.5\)); both annotators >92% correct relative to gold preference on hard 2-vs-3 cases.
  • On LongVideoBench, VideoJudge-7B achieves \(\Delta(\text{C-D}) = 1.16\) vs. 1.06/1.08 for Qwen2.5-VL 32B/72B, while most other video models score 0.20–0.83.
  • BERTScore of bootstrapped responses degrades monotonically: 91.1 (rating 4) → 86.9 (rating 1); BLEU 11.0 → 3.0, validating controlled quality gradation.

Limitations

  • Video is sampled at 1 fps (max 60 training / 180 evaluation frames), which may miss fine-grained temporal dynamics.
  • Dense video descriptions used as evaluator proxies in bootstrapping require inference from powerful models (Qwen2.5-VL-72B or GPT-4o-mini), incurring hidden computational cost.
  • Generator–evaluator disagreements concentrate around ratings 2–3, requiring more refinement iterations in the most ambiguous quality regime.
  • VideoJudgeR rubric model trained on only 10% of pointwise data due to computational constraints; full-data rubric training was not explored.
  • Several recent video MLLMs (VideoLLaMA3, VideoChat-Flash, SmolVLM2, Keye-VL) were excluded because they failed to follow judging instructions reliably, limiting generalizability analysis.

Relevance to Vision-Language Models

VideoJudge directly tackles the open evaluation bottleneck for video-language models: as VLMs scale to richer temporal reasoning tasks, automated metrics become increasingly inadequate and proprietary judge dependence reduces reproducibility. The finding that video tokens are strictly necessary for reliable judging—text descriptions and chain-of-thought alone are insufficient—has direct implications for designing evaluation pipelines in multimodal research. The bootstrapped rubric-generation capability moves MLLM evaluation toward interpretable, task-grounded assessment rather than opaque scalar outputs, which matters for alignment and capability analysis of VLMs. The efficient small-judge paradigm (7B matching 72B) offers a practical path for the community to evaluate large video VLMs at scale without proprietary APIs.