Skip to content

Stop Guessing When to Stop Testing: Efficient Model Evaluation with Just Enough Data

🕒 Published (v1): 2026-07-09 14:17 UTC · Source: Arxiv · link

Why this paper was selected

Adaptive benchmark sizing for model evaluation; directly actionable for rigorous VLM evaluation practice

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Fixed-size benchmarks waste compute by continuing evaluation past the point of statistical clarity, or conversely produce unreliable estimates when arbitrarily subsampled. This paper proposes an adaptive evaluation framework grounded in group sequential testing that stops evaluation exactly when user-defined statistical and practical criteria are met. Applied to the Open VLM Leaderboard (206 VLMs, 31 multimodal benchmarks), it achieves up to 80% reduction in evaluation cost while preserving statistical validity.

Problem

Current VLM/LLM benchmarking uses fixed sample sizes that cannot adapt to the actual evaluation objective—ranking, model selection, or development-time debugging each requires different statistical power. Heuristic subsampling (e.g., 1K examples) produces estimates with CI widths exceeding ±5.5 points and no statistical guarantees, while using full benchmarks wastes compute after diminishing returns set in. Neither approach aligns evaluation effort with the decision being made.

Method

The framework formalizes benchmark scoring as a mean-of-means over \(n\) datasets: $\(S_A(D) = \frac{1}{n} \sum_{i=1}^{n} \left( \frac{1}{m_i} \sum_{j=1}^{m_i} S_i^A(e_{ij}) \right)\)$ and frames pairwise comparison as a hypothesis test \(H_0: \delta_{A,B}(D) = 0\) vs. \(H_1: \delta_{A,B}(D) > 0\).

Evaluation proceeds in batches of size \(b\); at each stage \(k\) with cumulative size \(N_k = kb\), the test statistic is: $\(Z_k^{AB} = \frac{S_k^A(D) - S_k^B(D)}{\sqrt{\hat{\sigma}_k^2 \cdot (1/N_k)}}\)$

The Pocock spending function enforces a constant critical value \(c\) across all interim analyses so that the overall Type I error rate stays at \(\alpha\), correcting for the "peeking" problem. Six stopping rules are supported—efficacy, equivalence margin, precision-based (MDES), threshold crossing, futility, and diminishing returns—and can be combined. Implementation uses the gsDesign R package integrated into a Python pipeline, with defaults: \(b_\text{init}=600\), batch size 100, \(\beta=0.9\), Pocock function.

Key Contributions

  • Formal adaptive evaluation framework based on group sequential testing applied to LLM/VLM benchmarking
  • Six composable stopping rules covering the main practical evaluation objectives
  • Empirical analysis on Open VLM Leaderboard (206 models, 31 benchmarks) quantifying efficiency-reliability trade-offs for single-model scoring, pairwise comparison, and ranking
  • Demonstration that fixed heuristic subsampling (1.2K samples) reliably differentiates only 55% of model pairs at 95% confidence, vs. 76% with the adaptive framework

Results

  • 80% cost reduction when a ±2.5-point CI half-width is acceptable, with statistical guarantees maintained
  • 44% cost reduction under diminishing-returns stopping, sacrificing only 0.132 points in precision
  • ±3-point CI requires only ~15% of samples; ±2-point CI requires ~30%; ±1.5-point CI requires ~50%
  • For model pairs with performance gap >2 points, the framework saves ≥60% of evaluation effort
  • Fixed 1.2K-sample heuristic baseline: only 55% of 1K model pairs reliably differentiated at 95% confidence vs. 76% with adaptive stopping
  • Model pairs with gap <1.2 points typically require the full dataset under strict comparison (~24% of pairs)
  • At 8K examples (55% of data), CI width is 2.9 points; at full 14.4K, CI is 2.7 points—a 0.2-point gain at nearly double the cost

Limitations

  • Multiple-hypothesis correction for pairwise comparisons across many models is acknowledged but simplified away in experiments
  • The Pocock spending function requires pre-specifying the maximum number of interim analyses \(t\); mis-specification reduces power
  • Assumes independence of evaluation examples and approximate normality of \(Z_k^{AB}\) (CLT-justified, but may not hold for small batches or highly correlated datasets)
  • Relies on the gsDesign R package; integration overhead may deter adoption in pure-Python pipelines
  • Experiments use a single leaderboard snapshot (206 VLMs, commit dbc5e10); generalizability to other benchmark families (e.g., text-only LLM evals) is asserted but not empirically demonstrated in this paper

Relevance to Vision-Language Models

VLM evaluation is disproportionately expensive due to high-resolution image processing, long contexts, and LLM-as-Judge scoring—making adaptive stopping particularly impactful for this modality. The Open VLM Leaderboard is used as the primary testbed, directly addressing the community's pain point of ranking hundreds of VLMs across 31 multimodal benchmarks. The finding that heterogeneous multi-dataset benchmarks exhibit strong diminishing returns (driven by small, high-variance datasets) is a structural insight specific to multimodal evaluation design. Researchers tracking VLMs can use this framework to reduce the cost of ablation studies, checkpoint selection during fine-tuning, and leaderboard submissions.