Industrializing Prediction-Powered Inference: The GLIDE Library for Reliable GenAI and Agentic Systems Evaluation¶

🕒 Published (v1): 2026-05-29 13:10 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; prediction-powered inference for statistically valid agentic system evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Evaluating agentic systems requires either costly human annotation or biased LLM-as-judge proxies; Prediction-Powered Inference (PPI) statistically combines both into debiased estimates with valid confidence intervals. GLIDE is an open-source Python library that unifies state-of-the-art PPI estimators and samplers under a single scipy-style API, specialized to mean estimation. An agentic evaluation case study on R-Judge shows 16–20% narrower confidence intervals compared to labeled-only baselines at the same annotation budget.

Problem¶

Standard agentic system evaluation faces a dichotomy: full human annotation is unbiased but prohibitively expensive (especially for multi-step trajectories in high-stakes domains), while LLM-as-judge is cheap but produces systematically biased estimates without valid uncertainty quantification. Existing PPI methods address this gap theoretically, but their implementations are scattered across heterogeneous academic repositories with inconsistent notation, and no single library unifies the matching families of estimators and samplers needed for agentic evaluation workflows.

Method¶

GLIDE implements Prediction-Powered Inference for mean estimation \(\theta^\star = \mathbb{E}[Y]\), where \(Y\) is a human label and \(f(X)\) is a proxy (LLM-as-judge) prediction. The core PPI++ estimator is:

\[\hat{\theta}^\text{PPI++}_\lambda = \frac{1}{n}\sum_{i=1}^n Y_i + \lambda\left(\frac{1}{N}\sum_{j=1}^N f(X_j) - \frac{1}{n}\sum_{i=1}^n f(X_i)\right)\]

where \(\lambda^\star\) minimizes asymptotic variance, guaranteeing PPI++ is never worse than the labeled-only estimator. GLIDE decomposes the workflow into three independent stages — sampling (selecting which observations to annotate), annotation (out-of-scope human labeling), and estimation (debiased inference) — allowing samplers and estimators to be mixed and matched. A decision tree routes users to appropriate methods based on: availability of cost estimates (→ cost-optimal samplers), per-sample proxy uncertainty (→ active sampling), and natural strata (→ stratified estimators). The effective sample size \(n_\text{eff} = n \cdot \hat{\text{Var}}(\bar{Y}_n) / \hat{\text{Var}}(\hat{\theta}^\text{PPI++})\) serves as the headline metric for annotation savings.

Key Contributions¶

Unified library: Five PPI estimators (PPI++, Stratified PPI++, PTD, Stratified PTD, ASI) and four sampler families (uniform, stratified, active, cost-optimal) under a single scipy-style API
Reproducible validation suite: Monte Carlo coverage tests, sensitivity analyses to proxy quality, and effective-sample-size benchmarks across all estimators
Decision framework: Empirically-grounded decision tree for selecting sampler and estimator given available signals
Agentic evaluation case study: End-to-end demo on R-Judge benchmark with a real LLM-as-judge proxy, released as a reusable template notebook
Modular architecture: Samplers and estimators contributed as single-file modules implementing a common interface, lowering the barrier for community contributions

Results¶

Synthetic validation (PTD): At \(\rho = 0.9\) proxy-truth correlation, PTD achieves \(n_\text{eff} \approx 1100\) from 500 human labels + 1000 proxy labels — a 2.2× effective gain; at \(\rho = 0.1\), PTD matches the labeled-only baseline with no loss in coverage validity
R-Judge case study (\(n=100\) labeled, \(N=468\) proxy-labeled, \(\theta^\star \approx 0.525\), proxy bias \(\approx 13\) percentage points, \(\rho \approx 0.59\)):
Stratified PPI++ reduces 90%-CI width from \(\approx 0.164\) to \(\approx 0.131\) (20% reduction; \(n_\text{eff} \approx 157\))
ASI reduces width to \(\approx 0.135\) (18% reduction; \(n_\text{eff} \approx 148\))
Plain PPI++ reduces width to \(\approx 0.137\) (16% reduction; \(n_\text{eff} \approx 143\))
All four labeled-data protocols maintain nominal coverage across the full 0.55–0.95 confidence range; proxy-only baseline collapses well below target
Coverage robustness: PTD empirical coverage tracks 90% target across all \(\rho \in \{0.1, \ldots, 0.9\}\) regardless of proxy quality

Limitations¶

Mean estimation only: Quantiles, GLM coefficients, and general M-estimators are out of scope; ppi_py remains the reference for those estimands
Sample size floor: CLT-based estimators (PPI++, Stratified PPI++, ASI) require \(n \gtrsim 50\) labeled samples per stratum; below this threshold, bootstrap-based PTD variants are the fallback
Single proxy, i.i.d. assumption: Library does not support aggregating multiple proxies, anytime-valid constructions, or covariate/label shift between labeled and unlabeled sets
Annotation step unmanaged: Integration with labeling pipelines is left to the practitioner, which is often the most labor-intensive component in domain-specific deployments
Active sampling not strictly dominant: On R-Judge, stratification on a semantically meaningful low-cardinality axis (application domain) yields comparable or slightly larger gains than active sampling

Relevance to Agentic AI / LLM Agents¶

GLIDE directly addresses the evaluation bottleneck for agentic systems, where annotation costs per trajectory can be orders of magnitude higher than for single-turn LLM outputs, making principled annotation budget allocation critical. The library's stratified and active sampling components map naturally to the heterogeneous structure of agentic pipelines — different tools, sub-agents, and query types — enabling tighter evaluation at fixed budget by concentrating human review where the LLM-as-judge is least reliable. For researchers building or benchmarking LLM agents, GLIDE provides a statistically principled drop-in layer that sits downstream of evaluation orchestration frameworks (HELM, lm-eval-harness, RAGAS) and converts proxy scores into debiased estimates with distribution-free coverage guarantees. This is especially relevant as high-stakes agentic deployments (clinical, legal, financial) increasingly require auditable, uncertainty-quantified evaluation rather than point estimates from biased judges.