Skip to content

lmgame-Bench: How Good are LLMs at Playing Games?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Eric P. Xing/CMU; game-playing benchmark requiring perception, reasoning, planning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LMGame-Bench is a benchmark of six video games (Super Mario Bros, Tetris, Sokoban, Candy Crush, 2048, Ace Attorney) evaluated via a unified Gym-style API, with modular harnesses (perception, memory, reasoning) that can be toggled to isolate individual model capabilities. Evaluated across 13 frontier LLMs/VLMs, the benchmark remains unsaturated and exposes concrete failure modes in visual state extraction, spatiotemporal reasoning, and long-context planning. o3 and o1 top the leaderboard; current VLMs still struggle to extract board states directly from pixels—a task trivial for humans.

Problem

Existing game benchmarks either entangle multiple skills (vision, reasoning, memory), making failure attribution impossible, or are too easy (already near-solved) or too hard (no measurable progress). Direct "screenshot→action" evaluation collapses most models near random baselines, yielding insufficient discriminability among frontier models.

Method

LMGame-Bench wraps six games in a Gym-compatible interface, formalizing gameplay as a partially observable MDP \((S, A, R)\). Three harness modules are independently togglable: - Perception module: converts raw visual frames into structured textual representations (e.g., grid coordinates for Sokoban/Tetris/2048/Candy Crush; dialogue extraction for Ace Attorney; o3-generated descriptions for Mario). - Memory module: maintains a transient window of the last \(N\) (state, action, reward) tuples plus a reflection module that encodes explicit lessons from past failures to narrow the action space. - Reasoning module: toggles long chain-of-thought (long-CoT) for models that support extended reasoning traces.

Contamination is mitigated via (i) frame-reordering tests for vision-level contamination in Mario, and (ii) Sentence-BERT similarity checks plus entity masking/paraphrasing/enforced-reasoning prompts for text-level contamination in Ace Attorney. Prompt variance is reduced 33.8%–63.5% via DSPy's SIMBA optimizer using the canonical trajectory format \([\{J_{[\min(0,i-N):i-1]}\}, R_{i-1}, s_i] \mapsto a_i\).

Key Contributions

  • First benchmark to evaluate video games both with and without scaffolds, enabling controlled ablations of perception, memory, and reasoning in isolation.
  • Quantitative analysis linking game performance to core LLM capabilities via correlation and low-rank matrix factorization, revealing latent alignments between specific games and model capabilities.
  • Contamination detection and mitigation pipeline (vision-level frame reordering + text-level SBERT similarity + structured prompt mitigation).
  • DSPy SIMBA-based prompt standardization reducing cross-run variance by up to 63.5%.
  • Documented failure taxonomy: visual state extraction errors, invalid-move repetition in non-reasoning models, spatiotemporal misalignment (Mario), long-context information loss (Ace Attorney).

Results

  • Without harness, >75% of models score 0 on Sokoban and Ace Attorney; Tetris/Candy Crush scores are near random (\(\approx10.2\) and \(\approx116.5\) respectively).
  • With harness, 86.7% of game runs beat the random baseline (vs. 60% without); paired t-tests confirm significant improvement on Candy Crush, 2048, Tetris, Ace Attorney, Sokoban.
  • Rankings (harnessed): o3 > o1 > Gemini-2.5-Pro > Claude-3.7-Sonnet; among non-reasoning models, GPT-4.1 leads.
  • o3 harnessed Sokoban: \(8.0 \pm 2.8\) vs. human avg \(9.7 \pm 3.9\); o3 harnessed Tetris: 42 vs. human avg 353.3.
  • Claude-3.7-Sonnet (thinking) harnessed Candy Crush: \(484.0 \pm 53.7\) vs. human avg \(283.3 \pm 10.7\) (surpasses humans on this game).
  • Perception module yields largest gains for reasoning models in grid-based games (e.g., Gemini-2.5-Pro Sokoban: 0 → 4.3 harnessed).
  • Memory module most impactful for non-reasoning models in long-horizon games (2048, Candy Crush).
  • Prompt standardization reduces variance by 33.8%–63.5% in 2048.

Limitations

  • Harness modules partially remove the multimodal challenge: models receive text descriptions of game states rather than raw pixels, meaning harnessed VLM performance does not purely measure visual understanding.
  • Human baseline is small (\(n=3\) graduate students) with high variance (e.g., Mario: \(4333 \pm 2718\)), limiting statistical comparison.
  • Several evaluations use single runs due to API cost constraints (marked with * in tables), reducing reliability of those estimates.
  • Perception module for Mario uses o3 to generate textual descriptions, introducing a confound—o3's own visual capability affects all models using that module.
  • Text-level contamination mitigation (entity masking, paraphrasing) modifies the original game content, potentially changing task difficulty in ways not fully characterized.
  • No evaluation of open-source vision models (e.g., LLaVA, InternVL) beyond Llama-4-Maverick.

Relevance to Vision-Language Models

LMGame-Bench provides direct empirical evidence that current frontier VLMs fail at a task trivial for humans—extracting structured board state from screenshots—underscoring a fundamental visual grounding gap that persists even in reasoning-augmented models. The perception module ablation isolates whether failures stem from visual encoding versus downstream planning, which is a diagnostic methodology directly applicable to VLM evaluation in any grounded-reasoning context. The benchmark's finding that converting images to text representations substantially boosts even strong reasoning models suggests that VLM visual representations are not yet compositionally grounded enough for fine-grained spatial reasoning. This work complements VLM benchmarks focused on static image understanding by probing sequential, action-conditioned visual reasoning under partial observability.