MMBench-Live: A Continuously Evolving Benchmark for Multimodal Models¶

🕒 Published (v1): 2026-07-02 07:27 UTC · Source: Arxiv · link

Why this paper was selected

Continuously evolving VLM benchmark addressing contamination; directly relevant to agent evaluation infrastructure

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MMBench-Live is a continuously evolving multimodal benchmark built by a multi-agent automated pipeline that acquires temporally fresh real-world images, generates verifiable QA pairs, and maintains cross-version distributional alignment with the original MMBench. Each update produces ~5.9K evaluation instances in 1–2 hours at ~$30, achieving 96.06% manual answer correctness.

Problem¶

Static vision-language benchmarks become unreliable as VLMs are trained on ever-growing web-scale corpora: they suffer from temporal staleness, data contamination inflating reported scores, and prohibitive manual maintenance costs. Prior dynamic approaches—visual perturbations, LLM-based rewrites, or generation-based methods—introduce semantic drift, distributional gaps, or are tied to fixed data sources, making cross-version comparability difficult to preserve.

Method¶

The pipeline treats benchmark evolution as task-guided dataset construction in three stages:

Structured benchmark specification: MMBench is converted into a hierarchical template covering evaluation purpose, format, task hierarchy, and atomic tasks. For each atomic task, task-related visual patterns are identified by running a VLM over sampled instances across five semantic perspectives (visual content, visual style, OCR dependency, spatial relations, external knowledge), then consolidated by an LLM.
Feedback-controlled data acquisition: An acquisition planner generates a plan $P_t = \langle F_t, A_t, Q_t \rangle$ per atomic task. An executor collects images via three modes: source-specific structured crawling, keyword-guided Google Image API retrieval, and open-domain Flickr API sampling. A feedback controller applies a multimodal judge $J$ to evaluate each candidate against task purpose $G_t$ and visual patterns $B_t$, computing acceptance rate $\rho_q = \frac{1}{K}\sum_{i=1}^K a_i$ over top-$K=10$ results. If $\rho_q < \tau = 0.7$, rejected candidates' rationales are aggregated and fed back to update the query via $q_{m+1} = U(q_m \mid G_t, B_t, r)$, for at most $M=3$ iterations.
QA generation and executable verification: Multiple VLMs generate candidate triplets $z = (q, a, \pi)$ where $\pi$ is an executable solution plan. Tool-chain execution produces textual intermediates $z_k = T(o_k \mid x, z_{<k})$; a vision-blind LLM verifier then predicts $\hat{a} = \arg\max_y P(y \mid q, z_{1:K})$. A QA pair is accepted only if $a = \hat{a}$.

Key Contributions¶

MMBench-Live: a continuously evolving multimodal benchmark derived from MMBench with 5.9K fresh QA pairs per update cycle.
Distribution-consistent update strategy combining task-related visual pattern identification with feedback-controlled retrieval to preserve visual and semantic characteristics across versions.
Vision-blind executable verification that decouples correctness checking from direct image judgment, reducing hallucination in auto-generated QA.
Empirical demonstration that dynamic benchmark updates can maintain stable model rankings and weaker contamination-related memorization signals.

Results¶

QA correctness: 96.06% manual verification rate across all tasks (10% sample per task, 3 independent trials).
Construction cost: ~$30 and 1–2 hours per update vs. thousands to tens of thousands of dollars and 1–2 months for comparable manual pipelines.
Cross-version consistency (original MMBench vs. MMBench-Live):
Task-model level: Pearson $r = 0.52$ ($p = 1.15 \times 10^{-9}$), Spearman $\rho = 0.43$ ($p = 1.27 \times 10^{-6}$).
Model-averaged level: Pearson $r = 0.89$ ($p = 0.018$), Spearman $\rho = 0.83$ ($p = 0.042$) — strong rank preservation at aggregate level.
Model performance on MMBench-Live: Qwen2.5-VL-7B (83.10%) and DeepSeek-VL-7B-Chat (83.04%) top the leaderboard; older models (InstructBLIP: 73.85%, LLaVA-1.5: 74.83%) score substantially lower.
Contamination signal: MMBench-Live exhibits weaker PaCoST-based memorization confidence shifts than the original MMBench, suggesting reduced text-side contamination exposure.

Limitations¶

PaCoST contamination analysis is a proxy for text-side memorization only; it cannot directly detect visual leakage, metadata overlap, or web-crawl contamination.
Framework is instantiated solely from MMBench; generalizability to benchmarks with different task taxonomies or modalities is undemonstrated.
Feedback controller is hard-capped at $M=3$ iterations; tasks with very low initial alignment may still produce suboptimal retrieval queries.
Correctness rate (96.06%) measures final answer accuracy only, not validity of the executable solution plan $\pi$.
Construction cost is dominated by QA generation (~60%), tying scalability to API pricing of closed proprietary models (GPT-5 Mini, Gemini-3-Flash-Preview).

Relevance to Agentic AI / LLM Agents¶

MMBench-Live is explicitly constructed by a multi-agent pipeline in which specialized agents—acquisition planner, multimodal judge, feedback query updater, multi-model QA generators, tool-executing verifier, and vision-blind reasoner—coordinate to produce a reliable dataset, making it a concrete example of agentic tool use and feedback-driven self-correction at the data-production level. The feedback controller's iterative query refinement loop (observe → judge → aggregate rationales → update) exemplifies the plan-act-observe loop central to agentic system design. For researchers tracking LLM agent evaluation, this work demonstrates that live benchmark construction for complex multimodal tasks is tractable with current agent frameworks, and it raises the question of how well agent benchmarks themselves age—a concern mirrored in tool-use and coding agent evaluation suites. The executable reasoning verification pattern (tool chain → text intermediates → blind verifier) is directly transferable as a correctness-checking mechanism within agentic pipelines.