Skip to content

No-Human in the Loop: Agentic Evaluation at Scale for Recommendation

🕒 Published (v1): 2025-11-04 22:49 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ScalingEval is a multi-agent benchmarking framework that eliminates human annotators by using 36 LLMs as judges—orchestrated through specialized audit agents and majority-vote consensus—to evaluate complementary-item recommendation (CIR) at Walmart scale. It systematically compares GPT, Gemini, Claude, and Llama families across five product categories, producing reproducible ground-truth labels without any human in the loop. Key finding: Gemini-1.5-pro maximizes accuracy/coverage, Claude-3.5-sonnet maximizes decision confidence (~99%), and GPT-4o offers the best latency–accuracy–cost trade-off.

Problem

Evaluating LLM-based recommender systems at scale is prohibitively expensive with a single model and lacks reproducibility without human annotation. Traditional heuristics for complementary-item recommendation (category overlap, co-purchase statistics) cannot capture nuanced contextual relationships (e.g., "family-size vs. single-serve"), yet naïvely applying a single LLM across millions of item pairs is cost-prohibitive. No systematic benchmark existed to compare model families, sizes, and temperature settings as judges for this task.

Method

ScalingEval decomposes evaluation into a three-agent pipeline per model:

  1. CI Pattern Audit Agent: maps each anchor–recommendation pair \((item_a, item_r)\) to a set of complementary-item patterns \(P = \{p_i\}_{i=1}^M\), outputting \(out_{\text{pattern\_audit}} = \{(item_a, item_r): [p_j]\}\).
  2. Recommendation Issue Audit Agent: tags pairs with predefined issue codes \(S = \{s_i\}_{i=1}^K\), outputting \(out_{\text{issue\_audit}} = \{(item_a, item_r): [s_j]\}\).
  3. Report Generation Agent: chunks the dataset into batches of size \(k\), producing per-chunk reports \(r_i = \{t_i, c_i, n_i, P_i, I_i, q_i, a_i\}\) (total, complementary count, non-complementary count, patterns, issue codes, conflicts, agreement score), then aggregates to \(R = \{T, C, N, Q, P, I, A\}\) with constraint \(C + N = T\).

All 36 models independently run this pipeline. Scalable Majority-Vote Ground Truth Synthesis collects per-pair judgments into a decision matrix and applies majority voting with a 60% threshold over the top-25 quality-ranked models. Conflict resolution follows a strict priority ordering: Reject > Major > Minor > Good. The consensus report is selected for completeness and cross-dataset consistency.

Key Contributions

  • ScalingEval: a fully automated, no-human-in-the-loop multi-agent evaluation harness benchmarking 36 LLMs on CIR.
  • Consensus ground-truth synthesis via scalable majority voting over agentic crowdsourcing, eliminating human annotation cost.
  • Systematic cross-model comparison covering accuracy, confidence, coverage, Cohen's Kappa, issue-match, and latency/cost on 1,745 Walmart e-commerce pairs across 7 product categories.
  • Empirical identification of performance tiers and category-level difficulty (structured domains vs. lifestyle categories).
  • A reusable evaluation protocol template applicable beyond CIR to LLM lifecycle evaluation.

Results

Overall accuracy (Table 1): - Gemini-1.5-pro: 76.63% accuracy, 87.14% coverage — best overall - GPT-o1: 67.18% accuracy, 68.81% coverage — highest accuracy among GPT family, ~4.9× latency - GPT-4o: 60.13% accuracy, 64.63% coverage — best latency–accuracy–cost balance (1× baseline cost) - GPT-OSS-20B: 42.32% accuracy, 52.60% coverage — best open-source model - Claude-3.5-sonnet: 31.08% accuracy, 31.58% coverage — low overall accuracy but highest confidence

Confidence (Table 2, on determined pairs): - Claude-3.5-sonnet: 98.4–99.2% confidence, Cohen's Kappa 96.7–98.4% - Llama-3-8B-Instruct: 95.9–97.5% - GPT-4o: 93.1–95.1% - Gemini models: 87.9–88.6%

Category-level agreement rates: - High agreement: Sports & Outdoors (93.8%), Electronics (91.3%) - Low agreement: Clothing & Shoes (84.2%), Food & Beverages (85.4%)

Gemini-1.5-pro leads in 4 of 5 categories; GPT-o1 excels specifically in Clothing & Shoes.

Limitations

  • Dataset is limited to 1,745 pairs from Walmart's internal e-commerce platform; generalizability to other domains is unverified.
  • Ground truth itself is derived from LLM consensus, not human annotation — systematic model biases shared across families could corrupt the majority-vote labels undetected.
  • Conflict-resolution priority ordering (Reject > Major > Minor > Good) is manually defined and conservative; no ablation is provided on this design choice.
  • Claude-3.5-sonnet's very low coverage (~31%) is unexplained — high confidence on determined pairs masks that it abstains on ~69% of cases.
  • Temperature sensitivity analysis is limited to three settings (0.4, 0.6, 0.8); no analysis of prompt sensitivity or prompt-engineering effects.
  • Open-source models (Llama-3-3B/8B) perform poorly, but only small variants are tested.

Relevance to Harnesses / Meta-Harnesses

ScalingEval is a direct example of a meta-harness: it orchestrates multiple specialized sub-agents (pattern audit, issue audit, report generation) and aggregates their outputs through a formalized consensus mechanism, rather than relying on a single monolithic evaluator. The majority-vote synthesis layer is functionally analogous to a meta-harness arbitration layer that reconciles divergent agent outputs into a stable ground truth — a pattern directly applicable to any harness that must aggregate heterogeneous LLM judgments. For researchers building evaluation harnesses, this paper provides concrete design patterns: chunked batch processing for scale, tiered conflict resolution with explicit priority ordering, and the separation of pattern-audit from issue-audit responsibilities. The finding that cross-model confidence varies dramatically (77%–99%) while accuracy varies less sharply underscores the importance of harness-level aggregation to avoid over-relying on any single model's signal.