Skip to content

Orak: A Foundational Benchmark for Training and Evaluating LLM Agents on Diverse Video Games

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Foundational diverse video game benchmark for training and evaluating LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Orak is a benchmark for training and evaluating LLM agents across 12 real commercial video games spanning all 6 major genres, built on a plug-and-play Model Context Protocol (MCP) interface. It additionally releases a ~11k-sample fine-tuning dataset of expert LLM gameplay trajectories to adapt pre-trained LLMs into gaming agents, enabling systematic study of agentic modules, model comparisons, and knowledge transfer.

Problem

Existing game benchmarks for LLMs are insufficient for practical gaming-agent development on three axes: (1) they rely on text-only or 2D-grid games rather than complex real video games; (2) they do not systematically evaluate agentic modules—self-reflection, memory, planning, tool use—that are essential for complex gameplay; and (3) they provide no fine-tuning datasets to bridge pre-trained LLMs and effective game-playing agents.

Method

Orak wraps 12 popular commercial video games (Street Fighter III, Super Mario, Ace Attorney, Her Story, Pokémon Red, Darkest Dungeon, Minecraft, Stardew Valley, StarCraft II, Slay the Spire, Baba Is You, 2048) under a unified MCP interface. Each game environment and each agentic module (reflection, planning, memory, skill manager) runs as an independent MCP server exposing callable tools. At each step, the evaluator: (1) calls obs2text to convert game state to text, (2) invokes the configured agentic strategy (single module or chained sequence), and (3) calls text2act followed by step to advance the game. This decouples LLM backbones from both environments and agent strategies, enabling plug-and-play substitution.

The fine-tuning dataset is collected by running GPT-4o and o3-mini on all 12 games with reflection–planning–action chaining. Trajectories \(\mathcal{T} = \{\tau_1, \ldots, \tau_T\}\) are scored, and the top-scoring subset until \(|\{\tau\}| \geq 300\) per agentic module is selected, yielding ~11k samples total. Each sample \(\tau = \{(X^{a_i}, S, Y^{a_i})\}_{i=1}^n\) pairs a module prompt \(X^a\), game state \(S\), and LLM response \(Y^a\). Ten paraphrase augmentations per sample (via GPT-4o) increase linguistic diversity. Fine-tuning is supervised (SFT).

Game capabilities are annotated on seven dimensions—Rule Following (RF), Logical Reasoning (LR), Spatial Reasoning (SR), Long-text Understanding (LTU), Long-term Planning (LP), Error Handling (EH), Odds Handling (OH)—rated 1–3 by 8 human participants.

Key Contributions

  • First benchmark covering all 6 major video game genres with 12 real commercial games, supporting both LLM and VLM evaluation.
  • MCP-based plug-and-play interface decoupling game environments, agentic modules, and backbone LLMs for systematic, reproducible evaluation.
  • ~11k-sample SFT dataset of top-scored expert LLM gameplay trajectories with 10Ă— paraphrase augmentation, covering all 12 games under reflection–planning–action protocols.
  • Comprehensive evaluation suite: per-game score leaderboards, competitive Elo-based LLM battle arenas, and ablation studies on input modality, agentic strategy, and fine-tuning.
  • Seven-dimension principled capability taxonomy (RF, LR, SR, LTU, LP, EH, OH) for characterizing required LLM skills per game.

Results

  • Leaderboard (average rank across 12 games, lower = better): GPT-5 and Gemini-2.5-pro rank 3.5–3.6; GPT-4o ranks 3.6; o3-mini ranks 4.0; Claude-3.7 ranks 5.1; LLaMA-3.3-70B ranks 7.3; small open-source models rank 10–13.5.
  • Battle arena (Elo): GPT-4o achieves 1602.9; Gemini-2.5-pro achieves 1453.0; gap between proprietary and open-source models narrows in battle scenarios relative to absolute score leaderboard.
  • Agentic strategy ablation: Proprietary LLMs benefit measurably from extended agentic workflows (reflection + planning); open-source LLMs show limited gains.
  • Visual input ablation: Models across the board fail to extract sufficient value from visual game state inputs.
  • Fine-tuning transfer: SFT on Orak trajectories enables gameplay meta-knowledge to transfer from GPT-4o to LLaMA-3.2-1B/3B, with OOD game scores improving (e.g., LLaMA-3.2-1B OOD score rises to 16.7); generalization also observed on non-game benchmarks (math, web interaction).
  • Human novice average rank is not directly comparable but achieves ~22.7 average score vs. top LLM averages in the 5–10 range on many games.

Limitations

  • Fine-tuning dataset supports SFT only; reinforcement learning fine-tuning from live environment interaction is left for future work.
  • Open-source LLMs show minimal improvement from extended agentic workflows, suggesting the agentic module benefits are currently limited to high-capability proprietary models.
  • Visual input modality remains underutilized across all tested models.
  • Expert trajectories are generated by GPT-4o/o3-mini, so fine-tuning dataset quality is bounded by these teacher models and may embed their failure modes.
  • The paper does not cover multi-agent gameplay scenarios or human-level baselines across all games systematically.

Relevance to Agentic AI / LLM Agents

Orak directly targets a core challenge in agentic AI: assessing whether an agent's modular components—reflection, planning, memory, tool invocation—actually improve performance on complex, partially observable, long-horizon tasks. The MCP-based architecture mirrors how production agentic systems compose capabilities as callable tool servers, making Orak's design choices immediately transferable to real-world agent frameworks. The finding that fine-tuning on game trajectories generalizes to non-game tasks (math, web interaction) is significant for the broader agent training literature, suggesting that diverse interactive environments can serve as data flywheels for general-purpose agents. The demonstrated gap between proprietary and open-source models in leveraging agentic scaffolding highlights an under-studied axis of model capability relevant to anyone building or benchmarking LLM agent systems.