Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; game-based multimodal verifiable data boosts VLM general reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Game-RL proposes using video game environments as a scalable source of multimodal, verifiable RL training data for VLMs. The authors introduce Code2Logic, a pipeline that converts game code into structured VQA datasets with programmatically verified answers, and show that training exclusively on this game data transfers to out-of-domain general vision-language benchmarks.

Problem

Vision-language RL training has been confined to narrow domains (primarily geometry and chart reasoning), leaving the broader space of visual reasoning underexplored. Existing game-based VLM evaluations do not produce VQA-format training data, and scalable sources of diverse, programmatically verifiable multimodal reasoning supervision are lacking.

Method

Code2Logic is a three-step pipeline to synthesize verifiable game reasoning data: 1. Game code construction: LLMs (Claude 3.5, GPT-4o) generate playable game code that encodes state spaces and transition rules. 2. Task and QA template design: LLMs extract reasoning patterns from the game code and formalize them as question/answer templates spanning three task types — Target Perception, State Prediction, and Strategy Optimization. 3. Data engine construction: LLMs build a program that reuses core game functions (e.g., Sokoban's move()) to instantiate templates at scale with controllable difficulty (grid size, sequence length) via code parameters.

The resulting GameQA dataset covers 30 games, 158 tasks, and ~140K VQA questions across four cognitive categories: 3D Spatial Perception, Pattern Recognition, Multi-step Reasoning, and Strategic Planning. RL fine-tuning uses GRPO (Group Relative Policy Optimization) with a binary outcome reward (\(r=1\) if semantically correct, \(r=0\) otherwise), where semantic equivalence is judged by Qwen2.5-32B-Instruct-AWQ (validated at 100% accuracy on 300 manual checks). Difficulty is controlled along two axes: QA level (question complexity) and Plot level (image/grid complexity).

Key Contributions

• Code2Logic: a general method to convert game code into unlimited, programmatically verifiable VQA data with controllable difficulty.
• GameQA: 30 games, 158 tasks, ~140K questions across 4 cognitive categories; 20 games for training (in-domain), 10 held out for generalization testing.
• Demonstration that game-only RL training (5K samples) generalizes to 7 diverse out-of-domain vision-language benchmarks, competitive with domain-specific reasoning datasets (MAVIS, MultiMath, Multimodal-Open-R1).
• Scaling laws showing that increasing game diversity or data volume monotonically improves general VLM reasoning.

Results

• Qwen2.5-VL-7B: +2.65% average across 7 general benchmarks (MathVista 66.62→68.48, MathVerse 45.10→48.60, CharXiv 37.92→42.08, MMMU-Pro 36.32→39.99).
• InternVL2.5-8B: +2.60% average (MathVista 57.43→62.38, CharXiv 31.68→35.66).
• InternVL3-8B: +1.90% average (MathVista 68.72→73.24, MathVision 35.24→38.05).
• vs. general reasoning datasets (5K GameQA GRPO on Qwen2.5-VL-7B): GameQA-5K (+2.78 avg on out-of-domain games) outperforms MAVIS-8K (+0.52), Multimodal-Open-R1-8K (+1.24), MultiMath-8K (+1.29); on general benchmarks GameQA-5K achieves +2.37 vs. MultiMath-8K's +2.87.
• Combining GameQA-5K + MultiMath-8K yields +3.84 (out-of-domain games) and +3.29 (general benchmarks) over baseline.
• Scaling from 1K to 20K game samples shows a consistent upward trend across all 7 benchmarks (Figure 4).

Limitations

• Training used only 5K–20K samples; it is unclear whether further scaling saturates.
• 20 in-domain games may not cover all visual reasoning modalities (e.g., temporal reasoning, natural scene understanding).
• Game visuals are synthetic/2D-dominant; transfer to photorealistic or embodied scenarios is unconfirmed.
• Reward model (Qwen2.5-32B-AWQ) introduces a dependency on a large auxiliary model; its failure modes on novel answer formats are not fully characterized.
• Code2Logic requires manual verification of game code and data engine outputs, limiting full automation.

Relevance to Vision-Language Models

Game-RL directly addresses the data bottleneck in vision-language RL by providing a scalable, verifiable, and diverse alternative to domain-specific datasets, which has been a persistent constraint on VLM post-training. The finding that game-data RL transfers to benchmarks like MathVista, MMMU, and CharXiv implies that structured visual game environments elicit general-purpose spatial and relational reasoning skills transferable across modalities. The Code2Logic paradigm is extensible — any game with explicit transition rules can be converted — making this a potentially high-leverage direction for constructing RL training corpora as geometry and chart data saturate. For VLM researchers, the scaling results (diversity × volume → general reasoning) add empirical weight to the hypothesis that breadth of training distribution, not domain match, drives robust multimodal reasoning.