Skip to content

Credit-Budgeted ICPC-Style Coding: When Agents Must Pay for Every Decision

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Credit-budgeted coding benchmark measures real-world agent efficiency tradeoffs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

USACOArena is an ICPC-style competitive coding arena that imposes a unified "credit" economy on coding agents, charging for every LLM inference token, local test, and elapsed second. It reveals that frontier agents (Gemini-2.5-pro, GPT-5-Codex) fail to optimally balance accuracy with resource consumption, exhibiting divergent, path-dependent strategies. The benchmark reframes agent evaluation from static correctness to cost-aware, dynamic resource management.

Problem

Existing coding benchmarks (SWEbench, USACO, etc.) evaluate agents in a "temporal vacuum"—measuring only final code correctness while ignoring API token costs, wall-clock time, and compute overhead. As multi-agent swarms scale, this cost-blindness risks catastrophic budget exhaustion in real deployments. No prior framework forces agents to make strategic trade-offs between exploration, exploitation, and resource conservation under strict budget constraints.

Method

USACOArena wraps the USACO problem corpus (48 problems from the 2024–2025 season, Bronze–Platinum difficulty, 12 per match) in an ACM-ICPC-style arena governed by a unified credit budget \(C_{\text{limit}}\). The agent policy \(\pi\) must satisfy:

\[\max_{\pi} \text{Score}(\pi) \quad \text{subject to} \quad C_{\text{action}}(\pi) + C_{\text{time}}(\pi) \leq C_{\text{limit}}\]

Credits are depleted by three sources: (1) LLM inference cost normalized by API pricing, (2) hint/test costs for local debugging runs, and (3) delivery time cost \(\alpha T\) where \(T\) is absolute wall-clock time and \(\alpha\) is an adjustable coefficient. Wrong-answer submissions incur penalty credits \(C_{\text{penalty}}\) used as a tie-breaker. Communication between the arena server and agents uses the Model Context Protocol (MCP) in a turn-based JSON loop; code is executed in a sandboxed online judge. A "Living Benchmark" strategy—ingesting new USACO seasons annually—mitigates data contamination.

Key Contributions

  • Credit economy framework: formalizes agent evaluation as a Budget-Constrained POMDP, unifying tokens, time, and test costs into one depleting budget signal.
  • USACOArena platform: interactive MCP-based arena with a secure sandbox, reproducible protocols, and support for single agents and multi-agent swarms.
  • Strategic profiling of frontier models: reveals that Gemini-2.5-pro (aggressive exploration, breadth-first) and GPT-5-Codex (conservative exploitation, near-perfect first-submit accuracy) achieve divergent win rates (70% vs. 30%) despite GPT-5-Codex having a higher peak score (29 vs. 19).
  • Self-play emergent behavior analysis: identical Gemini-2.5-pro agents produce highly variable outcomes via path-dependent "First-Move Effects" (Success Loop vs. Panic Loop), validating the environment's suitability for RL-based agent training.

Results

  • Gemini-2.5-pro and GPT-5-Codex form a stable two-tier hierarchy across all four 2024–2025 USACO contests; all other agents show volatile, often zero scores.
  • Gemini-2.5-pro avg. rank: \(1.3 \pm 0.47\), win rate 70%; GPT-5-Codex avg. rank: \(1.7 \pm 0.47\), win rate 30%.
  • GPT-5-Codex peak score: 29; Gemini-2.5-pro peak score: 19 — yet Gemini-2.5-pro wins more often due to broader problem coverage.
  • Current top agents score ~15 points against a theoretical maximum of 54, placing them in the "Gold/Advanced" tier (comparable to strong high-school competitors); the Platinum/Expert tier (>24) is not consistently reached.
  • On the hardest contest (USACO 2025 US Open), Gemini-2.5-pro scores 14.6 vs. GPT-5-Codex's 3.0.
  • Self-play across 18 head-to-head matches between identical Gemini-2.5-pro instances produces high score variance with no simple correlation between credit consumed and final score.
  • Neither single agents nor agent swarms can currently optimally balance accuracy with strict credit limits.

Limitations

  • Cross-model benchmarking sets \(\alpha = 0\) (ignoring wall-clock time) to avoid confounding network latency differences between commercial APIs, leaving the time-cost dimension partially unevaluated in the main results.
  • Evaluation is restricted to competitive programming (USACO), which offers clean problem specifications; generalization to messier real-world software engineering tasks is unverified.
  • The agent prompts are non-prescriptive by design (strategies must be emergent), but prompt sensitivity ablations are mentioned as ongoing (section 4.4 was truncated), leaving robustness to prompt variation incompletely reported.
  • Only two agents (Gemini-2.5-pro, GPT-5-Codex) consistently clear qualification; the benchmark may be too hard to meaningfully differentiate most current models.
  • Multi-agent swarm experiments are described as "early implementations" with limited depth in the provided text.

Relevance to Agentic AI / LLM Agents

USACOArena directly addresses a critical blind spot in agentic evaluation: current benchmarks reward agents for what they produce, not how efficiently they produce it. By treating resource consumption as a first-class signal, it operationalizes the metacognitive challenge central to real agent deployment—when to explore, when to exploit, and when to stop. The self-play experiments expose that even a single fixed policy produces wildly divergent trajectories (Success Loop vs. Panic Loop), providing the reward gradient necessary for RL-based agent training. This work connects to the broader shift from single-turn LLM evaluation toward multi-step, economically grounded agent benchmarks, and is directly relevant to anyone building or evaluating agent swarms where shared compute budgets must be managed.