Skip to content

The PokeAgent Challenge: Competitive and Long-Context Learning at Scale

🕒 Published (v1): 2026-03-16 17:25 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PokéAgent Challenge is a two-track benchmark built on Pokémon's battle simulator and RPG game, targeting partial observability, game-theoretic reasoning, and long-horizon planning simultaneously. It releases the first open-source multi-agent orchestration harness for RPG speedrunning and a 22M+ trajectory dataset. A key empirical finding: no frontier LLM achieves non-trivial RPG progress without a sophisticated harness.

Problem

Prior LLM-plays-Pokémon demonstrations (Claude, Gemini, GPT) used different games, different harnesses, and different evaluation criteria, making cross-system comparison impossible and conflating model capability with harness design. No standardized benchmark stressed partial observability, adversarial reasoning, and long-horizon planning simultaneously at scale.

Method

Two complementary evaluation tracks share a common infrastructure:

Battling Track — two-player zero-sum Pokémon Showdown battles with imperfect information; agents ranked via Full-History Bradley–Terry (FH-BT) rating. LLM agents use a harness wrapping game state as structured text with configurable depth-limited minimax search. RL agents are trained on 4M human replays + 18M self-play trajectories.

Speedrunning Track — episodic MDP \(\mathcal{M} = (S, A, T, R, \gamma)\) over Pokémon Emerald with \(R = +1\) per milestone and \(\gamma = 1\). Agents receive visual frames plus limited state info (party, HP, status). The PokéAgent baseline is a multi-agent orchestration system coordinating MCP tools (A* pathfinding, button inputs, knowledge retrieval) with specialized sub-agents (battle, reflection, gym puzzles, objective verification), a central orchestrator maintaining a route plan, and automatic context compaction for multi-thousand-step episodes.

To attribute performance, the paper introduces a harness × model evaluation matrix decomposing systems along five dimensions: state representation \(S\), tools \(T\), memory \(M\), feedback \(F\), and fine-tuning \(\Phi\).

Key Contributions

  • Two-track standardized evaluation framework (Battling + Speedrunning) with public leaderboard at pokeagentchallenge.com
  • Largest public Pokémon battle dataset: 4M human demonstration trajectories + 18M synthetic self-play trajectories + 200K+ curated teams
  • First open-source multi-agent orchestration system for long-horizon RPG play (PokéAgent harness with MCP tools and sub-agents)
  • Harness × model attribution framework (S, T, M, F, Φ) enabling fair cross-paradigm comparison
  • NeurIPS 2025 competition results (100+ teams, 100K+ battles) plus BenchPress orthogonality analysis showing Pokémon battling is near-orthogonal to 49 standard LLM benchmarks

Results

  • RL vs. humans (Battling): Strongest RL baselines competitive with top-500 human players (~90th percentile GXE) in Gen 1 OU and Gen 9 OU; current ceiling remains sub-superhuman
  • LLM Battling (GXE on self-play ladder): Gemini 3.1 Pro 91%, GPT-5.2 90%, Claude Opus 4.6 70%, Claude Sonnet 4.6 55%; heuristic bot only 13%
  • Speedrunning baselines (PokéAgent harness, first gym): Gemini 3 Flash achieves fastest mean completion ~2:24h; Claude Sonnet 4.5 completes all 15 milestones but with high variance (6:25–20:45); GPT-5.2 intermediate; best organizer baseline is ~1.8× slower than average human (1:22:05)
  • Harness ablation: CLI-agent harnesses (Claude Code, Codex CLI, Gemini CLI) struggle to maintain coherence over thousands of sequential RPG decisions vs. PokéAgent harness
  • NeurIPS competition Speedrunning: 6/22 valid-submission teams achieved 100% completion; winner (Team Heatz) used Scripted Policy Distillation (SPD) — LLM decomposes task into sub-goals
  • NeurIPS Battling: 13/16 qualifying teams extended the RL baselines; #1 seeds PA-Agent (Gen 1) and FoulPlay (Gen 9) won their brackets

Limitations

  • Speedrunning evaluation scoped to only the first gym (Roxanne); full-game evaluation deferred until performance saturates at current scope
  • Even the best harness-wrapped baseline is ~1.8× slower than average human and far from the human world record
  • Battle rulesets limited to Gen 1 OU and Gen 9 OU at release; other formats supported but not yet the focus
  • Self-play ladder GXE figures for LLMs are measured within an isolated self-play pool, not against the full human ladder
  • Context compaction approach for long episodes is necessary but introduces potential information loss over thousands of steps

Relevance to Harnesses / Meta-Harnesses

This paper is directly a harness research artifact: it empirically demonstrates that harness design, not model capability, is the primary determinant of success in long-horizon sequential tasks — no frontier model achieves non-trivial RPG progress without a sophisticated harness. The 5-dimension attribution framework \((S, T, M, F, \Phi)\) provides a reusable vocabulary for decomposing and comparing harness designs, directly applicable to any meta-harness evaluation effort. The open-source PokéAgent orchestration system (MCP tools + specialized sub-agents + context compaction + central orchestrator) is a concrete reference implementation of modular, reproducible harness architecture. The comparison of generic CLI-agent harnesses (Claude Code, Codex CLI, Gemini CLI) against a domain-adapted orchestration harness quantifies the performance gap attributable to harness specialization rather than model choice.