MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models

🕒 Published (v1): 2026-07-02 07:27 UTC · Source: Arxiv · link

Why this paper was selected

MMBench-Live; continuously evolving benchmark combats contamination — raises evaluation rigor bar

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMBench-Live is a continuously evolving multimodal benchmark that replaces static VLM evaluation sets with a multi-agent automated pipeline generating temporally fresh QA instances. Each update produces ~5.9K new evaluation pairs in 1–2 hours at ~$30, while preserving distributional alignment and cross-version model rankings with the original MMBench. The framework addresses data contamination by restricting image retrieval to a one-year recency window and using executable reasoning verification rather than direct visual judgment.

Problem

Static multimodal benchmarks suffer from three compounding failures: (1) temporal staleness as models train on ever-growing web corpora, (2) data contamination where benchmark samples leak into pretraining data inflating scores (e.g., >16% of MMLU contaminated in LLaMA-2), and (3) high maintenance cost making frequent updates impractical. Prior dynamic approaches—visual perturbations, linguistic rewrites, or generation-based synthesis—degrade semantic alignment or introduce distributional gaps. Real-world data pipelines exist but are tied to fixed sources and lack cross-version comparability guarantees.

Method

The pipeline has three stages, all instantiated from a structured benchmark specification (evaluation purpose, format, task hierarchy, atomic tasks):

1. Task-Related Visual Pattern Identification (Algorithm 1): For each atomic task \(t\), a VLM analyzes a sampled subset \(E'_t\) across five semantic perspectives (visual content, style, OCR dependency, spatial relations, external knowledge), producing pattern clusters \(C_t\). An LLM consolidates these into a compact pattern set \(B_t\) that implicitly constrains valid instances.

2. Feedback-Controlled Data Acquisition (Algorithm 2): Three retrieval modes (structured crawl, keyword-guided Google Image API, open-domain Flickr stream) are managed by an acquisition planner. A multimodal judge \(J\) evaluates each candidate \((x_i, c_i)\) against task purpose \(G_t\) and visual patterns \(B_t\), outputting binary acceptance \(a_i\) with rationales \(r_i\). The acceptance rate \(\rho_{q_m} = \frac{1}{K}\sum_{i=1}^K a_i\) drives query refinement: if \(\rho_{q_m} < \tau = 0.7\), rejected rationales are aggregated into feedback \(r\) and a query update \(q_{m+1} = U(q_m \mid G_t, B_t, r)\) is issued via LLM, for up to \(M=3\) iterations.

3. QA Generation and Executable Reasoning Verification: Multiple VLMs generate candidate triplets \(z = (q, a, \pi)\) where \(\pi = (o_1, \ldots, o_K)\) is an executable solution plan. Tools (VisionReasoner for detection/segmentation, Depth Anything V2 for depth, Qwen2.5-VL/GOT-OCR-2 for OCR) execute the plan sequentially: \(z_k = T(o_k \mid x, z_{<k})\). A vision-blind text verifier then predicts \(\hat{a} = \arg\max_y P(y \mid q, z_{1:K})\). The pair is accepted only if \(a = \hat{a}\), decoupling validation from direct visual hallucination risk.

Key Contributions

• MMBench-Live: a continuously evolving multimodal benchmark with 5.9K instances, ~$30/update, 1–2 hour turnaround, 96.06% manual answer correctness rate.
• Distribution-consistent update strategy: task-related visual pattern identification guides data collection to preserve task semantics and visual characteristics across versions.
• Feedback-controlled acquisition loop: lightweight query refinement without model fine-tuning or human annotation improves semantic alignment of retrieved images.
• Executable reasoning verification: vision-blind validation via tool-executed solution plans reduces hallucination-induced annotation errors.

Results

• Model performance (Table 1): Qwen2.5-VL-7B-Instruct achieves the highest overall score (83.10%), slightly above DeepSeek-VL-7B-Chat (83.04%); older models (InstructBLIP-Vicuna-7B, LLaVA-v1.5-7B) score 73–75%.
• Cross-version consistency (Table 2): Model-averaged Pearson \(r = 0.8876\) (\(p = 0.018\)) and Spearman \(\rho = 0.8286\) (\(p = 0.042\)) between MMBench and MMBench-Live; task-model level correlations are weaker (\(r = 0.52\), \(\rho = 0.43\)) but statistically significant.
• QA correctness: 96.06% average answer correctness across tasks via manual verification (10% sample, 3 independent trials).
• Contamination signal: MMBench-Live exhibits weaker PaCoST-based memorization confidence shifts than original MMBench under Qwen2.5-VL-7B-Instruct evaluation.
• Cost reduction: Orders-of-magnitude cheaper and faster than manual pipelines (thousands of dollars, 1–2 months).

Limitations

• PaCoST measures text-side memorization only; visual leakage, metadata overlap, or web-exposure contamination cannot be directly diagnosed.
• The 96.06% correctness rate is measured on answers only, not on the validity of the full executable solution plan \(\pi\).
• Retrieval is dependent on Google Image API and Flickr API availability; changes to these services could disrupt the pipeline.
• The feedback controller runs at most \(M=3\) query refinement iterations; poorly specified tasks may still yield misaligned samples after exhausting the budget.
• Evaluation is limited to 7B-scale open-source models; behavior at larger model scales is not characterized.

Relevance to Vision-Language Models

MMBench-Live directly addresses a central challenge in VLM evaluation: the rapid obsolescence of static benchmarks as pretraining data scales and model capabilities improve. For researchers tracking VLMs, this work provides both an evolving evaluation resource and a reusable construction framework applicable to any benchmark with a structured task hierarchy. The executable reasoning verification approach is particularly relevant to ongoing work on reducing hallucination in VLM-generated data, showing that decoupling visual perception from correctness checking via tool-executed plans improves annotation reliability. The contamination analysis also provides a practical methodology for comparing benchmark robustness, complementing efforts on benchmark integrity in the broader VLM evaluation literature.