PACE: A Proxy for Agentic Capability Evaluation¶

🕒 Published (v1): 2026-07-02 10:59 UTC · Source: Arxiv · link

Why this paper was selected

PACE: cheap proxy replacing expensive SWE-Bench/GAIA evals — major practical win for agent builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PACE is a framework that selects a compact subset (~100 instances) of cheap, non-agentic benchmark items (instruction following, code generation, tool calling, etc.) whose aggregate scores reliably predict performance on expensive agentic benchmarks (SWE-Bench, GAIA). At equal predictive quality, PACE costs roughly 100× less than running random subsets of the target agentic evaluation itself.

Problem¶

Full agentic benchmark evaluations (SWE-Bench Verified, GAIA, WebArena) require complex infrastructure, hours-to-days of execution, and can cost thousands of dollars per model run. This makes rigorous agent evaluation disproportionately accessible to well-resourced groups and forces practitioners to evaluate less frequently or on limited subsets. The predictive relationship between cheap non-agentic benchmarks and agentic task performance is not well understood.

Method¶

PACE frames proxy construction as a budget-constrained subset-selection problem over a pool of 19 non-agentic benchmarks covering 11 capability dimensions. Given a budget of \(C\) proxy instances and a calibration set of models with known scores on both source and target benchmarks:

Instance scoring: Two filter-based signals are computed for each candidate instance \(i\):
Target relevance (local): \(|\rho_i| = |\text{Spearman}(X_{M_\text{train}, i}, \bar{y}_{M_\text{train}})|\)
Geometric importance (global): leverage score \(h_i = \sum_c V_{c,i}^2\) from a thin SVD \(X = U\Sigma V^\top\)
Dual selection: A local subset \(L\) picks top-\(C_L\) by \(|\rho_i|\) alone; a global subset \(G\) picks top-\(C_G\) by the product \(h_i \times |\rho_i|\). The union \(P = L \cup G\) with \(|P| = C\) is the proxy.
Regression: For Goal A (performance prediction), a linear least-squares model \(\hat{y}_m = w^\top x_m\) is fit on calibration models, with target-instance bootstrap resampling to reduce label noise. For Goal B (pairwise preference), a logistic Bradley-Terry model predicts which of two models scores higher: \(\hat{b}_{m,m'} = \mathbb{1}[w_g^\top(x_m - x_{m'}) > 0]\).
Ensemble: Final prediction \(\hat{y} = \lambda \hat{y}_L + (1-\lambda)\hat{y}_G\), with \(\lambda\) and the budget split \((C_L, C_G)\) tuned by held-out validation.

Evaluation follows strict leave-one-out cross-validation (LOOCV) over 14 models.

Key Contributions¶

PACE framework: A principled, two-signal instance-selection method (SVD leverage + Spearman correlation) for building non-agentic proxy benchmarks that predict agentic performance.
PACE-Bench: A concrete 100-instance proxy spanning 19 source benchmarks, evaluated against GAIA, SWE-Bench Verified, SWE-Bench Multimodal, and SWT-Bench.
Predictable cost-accuracy tradeoff: Quality improves smoothly with proxy budget \(C\) and saturates around \(C = 100\), enabling informed resource allocation.
Interpretability: The selected instance distribution reveals which capabilities (e.g., planning for SWE-Bench Verified, verification/IF for GAIA) are most predictive per agentic target.
Generalization: Instance selection on a training set of models generalizes to held-out models under LOOCV, validating deployment relevance.

Results¶

Goal A (performance prediction) with \(C = 100\), LOOCV across 14 models and 4 benchmarks:
Average MAE: 3.80% (range: 2.09% for SWE-Bench Verified to 5.77% for GAIA)
Average Spearman correlation: 0.807 (range: 0.67–0.89)
Average Pearson correlation: 0.74
Goal B (pairwise preference) with same proxy:
Average pairwise accuracy: 84.37% (range: 78.54%–90.11%), vs. 50% random baseline
Cost efficiency: PACE matches random target-subsampling quality at roughly 1/100 the cost in USD across all three metrics (MAE, Spearman, pairwise accuracy) and all four agentic targets (Figure 1).
Capability analysis: Instruction following and reasoning are universally selected (100/100 instances); planning and code dominate for SWE-Bench Verified; multimodal understanding is prominent for SWE-Bench Multimodal; verification and test dominate SWT-Bench.

Limitations¶

Calibration requires a set of models with known scores on both source and target benchmarks; small calibration sets (here |M| = 14) introduce regression noise.
LOOCV with 14 models is a thin evaluation pool; performance on models architecturally different from calibration models is untested.
The framework predicts mean benchmark scores, not per-instance outcomes; it does not replace agentic evaluation for fine-grained failure analysis.
Source benchmark pool (19 benchmarks) may not cover all capability dimensions that future agentic tasks require, limiting forward generalization.
Agentic benchmark scores themselves are noisy due to environment variability and small instance counts (~100–500), which sets a ceiling on proxy accuracy.

Relevance to Agentic AI / LLM Agents¶

PACE directly addresses a key infrastructure bottleneck in agentic AI research: the prohibitive cost of evaluating LLM agents at scale. By showing that non-agentic atomic capability scores are predictive of complex agentic outcomes (SWE-Bench, GAIA), it validates the capability-decomposition view of agent performance and provides a practical tool for fast model selection, routing, and development iteration. The interpretability analysis—which capabilities drive which benchmarks—is directly actionable for researchers designing agent scaffolds or training curricula, since it maps agentic task demands to measurable atomic skills. For practitioners, PACE-Bench enables frequent model comparisons during training runs without committing to full agent rollouts.