Skip to content

KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

KORGym is a dynamic, game-based evaluation platform with 50+ text and multimodal games designed to assess LLM and VLM reasoning capabilities independent of pretraining knowledge. It supports multi-turn interaction, configurable difficulty, and reinforcement learning integration. Experiments on 19 LLMs and 8 VLMs reveal consistent intra-family reasoning profiles and a systematic performance gap between open- and closed-source models.

Problem

Existing LLM benchmarks are either domain-specific (e.g., AIME, PHYBench), entangled with pretraining memorization, or limited to single-turn evaluation. Prior game-based platforms either introduce opponent noise that confounds pure reasoning measurement (TextArena, SPIN-Bench) or lack robust RL support and multimodal coverage (LogicGame, gg-bench). No prior platform provides a comprehensive, knowledge-orthogonal, multi-turn, multimodal reasoning testbed with RL integration.

Method

KORGym builds on the knowledge-orthogonal reasoning philosophy of KOR-Bench and the Gymnasium RL interface. It comprises four modules: (1) Inference Module for asynchronous model inference and checkpointing; (2) Game Interaction Module with standardized generate, print_board, and verify APIs; (3) Evaluation Module for score computation; (4) Communication Module for inter-module messaging.

Games span six reasoning dimensions: mathematical/logical, control interaction, puzzle, spatial/geometric, strategic, and multimodal reasoning. Scoring uses three rules—binary (win/loss), proportional (fraction correct), and cumulative (incremental)—aggregated via a Capability Dimension Aggregated Mean that log-compresses scores exceeding [0,1], min-max normalizes per game across all models, then averages within each reasoning dimension. Data leakage is quantified via a Knowledge Impact Coefficient β measuring how much a model's performance degrades when game rules are withheld.

Key Contributions

  • Suite of 50+ text- and vision-based games spanning six reasoning dimensions, including 9 multimodal games.
  • KORGym framework with RL environment support, parameterized difficulty, and standardized APIs enabling reproducible multi-turn evaluation.
  • Capability Dimension Aggregated Mean metric for fair cross-game, cross-dimension score aggregation.
  • Knowledge Impact Coefficient β to quantify pretraining data leakage per game/model.
  • Comprehensive empirical analysis of 19 LLMs and 8 VLMs, with ablations on modality, reasoning paradigm, RL, and response length.

Results

  • Overall LLM ranking (Avg %): O3-mini 82%, Gemini-2.5-Pro 79%, O1-2024-12-17 73%, Doubao-1.5-thinking-pro 72%, DeepSeek-R1 71%; best open-source instruction model (Qwen3-32B-thinking) reaches 60%.
  • VLM multimodal tasks: Gemini-2.5-Pro leads across most games (e.g., Circle the Cat 15%, Bubble Ball Sorting 90%, Wordle 85%); Doubao-vision-250115 and InternVL3-78B are competitive on some tasks; most models score 0% on Square Addition.
  • Modality effect: Textual game scores consistently exceed visual scores for open-source VLMs; some closed-source VLMs (notably Gemini) score higher on visual versions than textual versions.
  • Thinking vs. non-thinking: Thinking models uniformly outperform size-matched non-thinking variants (e.g., DeepSeek-R1-Distill-Qwen-32B > Qwen2.5-72B-Instruct).
  • Reasoning paradigm ablation: Disabling individual paradigms (code, math, algorithm) paradoxically increases average scores for most models, attributed to overreliance on memorized patterns.
  • Response length: Strong positive correlation between token count and KORGym score, with diminishing returns at high lengths.
  • RL benefit: Doubao-1.5-thinking-pro (RL-trained with DAPO/VAPO) achieves 0.84 in puzzle reasoning, surpassing O1 and O3-mini in that dimension.

Limitations

  • The Knowledge Impact Coefficient β is estimated via ablation (removing game rules), not a direct measurement of training data overlap, so leakage estimates are indirect.
  • Only 8 VLMs evaluated on multimodal games; VLM coverage is narrower than LLM coverage.
  • Multimodal games number only 9 of 50+, limiting the depth of visual reasoning assessment.
  • Zero-shot evaluation only; few-shot or chain-of-thought prompting variants are not studied.
  • Leaderboard relies on closed-source API calls, introducing potential temporal variability as models are updated silently.

Relevance to Vision-Language Models

KORGym provides one of the first game-based multi-turn evaluation environments that includes dedicated multimodal tasks (9 games spanning jigsaw puzzles, map simulation, circle-the-cat, etc.) evaluated on 8 VLMs, surfacing a concrete finding: open-source VLMs exhibit significantly weaker visual-spatial grounding than text-based reasoning, while frontier closed-source VLMs (Gemini-2.5-Pro) sometimes surpass their text-only performance on visual tasks, suggesting qualitative differences in multimodal integration. The knowledge-orthogonality design principle—minimizing pretraining data leakage—is particularly valuable for VLM evaluation because it decouples visual perception from memorized factual associations. The RL-compatible environment and parameterized difficulty also open a path for training VLMs on game-based multimodal tasks, directly relevant to embodied and agentic VLM research.