Skip to content

Evaluating LLMs in Open-Source Games

🕒 Published (v1): 2025-11-29 07:46 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically studies LLM agents playing open-source games — game-theoretic settings where players submit inspectable Python programs instead of direct actions — across the Iterated Prisoner's Dilemma and Coin Game. It introduces SPARC, a strategic code classification benchmark, and shows that LLMs can reason about opponent programs, develop sophisticated counter-strategies, and converge to approximate program equilibria with both cooperative and deceptive behaviors.

Problem

Prior work on LLMs in game-theoretic settings studied agents producing direct actions (opaque policies). Open-source game theory — where policies are explicit, inspectable programs — was previously limited to theoretical analysis because suitable code-generation systems didn't exist. There was no empirical characterization of whether LLMs can (a) reason about strategic code and (b) develop and sustain cooperative or deceptive program equilibria in repeated play.

Method

Three components:

  1. SPARC benchmark: 239 IPD strategies from the Axelrod Python library, evaluated under three perturbation levels (unmasked, class-name-masked, fully obfuscated via AST identifier renaming). LLMs must classify whether a strategy cooperates against a pure Cooperator for 10 rounds. Zero-shot and chain-of-thought prompting compared across 10+ open/closed/reasoning models.

  2. Dyadic repeated open-source games: Kimi-K2 agents (10 seeds × 10 meta-rounds) play IPD and Coin Game. Each meta-round: agent writes a natural-language spec then Python strategy; programs exchange source code before execution. Three agent types tested: Cooperative Payoff Maximization (CPM), Deceptive Payoff Maximization (DPM), Payoff Maximization (PM). Strategic features measured via GPT-4o judge (counter-measure, exploitation attempt, feint, direct imitation, independent development) and syntactic metrics (Cyclomatic Complexity, Halstead Effort, Opponent Script Access Score via taint analysis).

  3. Evolutionary analysis: Replicator dynamics on a payoff matrix derived from all pairwise matchups of CPM/DPM/PM; flow fields on a 2-simplex identify evolutionary stable equilibria.

Key Contributions

  • SPARC: first benchmark for strategic code classification, testing LLMs' ability to predict cooperative behavior of IPD programs under obfuscation.
  • First empirical study of LLM behavior in open-source (program) games.
  • Characterization of emergent strategic mechanisms (counter-measures, feints, exploitation attempts) by agent objective type.
  • Evolutionary analysis showing game-dependent equilibria: PM dominates in Coin Game; CPM and DPM both stable in IPD.

Results

  • SPARC: o4-mini achieves 87.6% (unmasked ZS), 88.0% (masked ZS), 84.2% (obfuscated ZS) — top overall. DeepSeek-V3 reaches 86.3%/87.6% (COT), GPT-4.1 reaches 85.1%/83.8% (COT). Chain-of-thought significantly improves all non-reasoning models (e.g., Mistral Small: 40.2% → 79.7% on unmasked). Obfuscation causes only minor accuracy drops. Stochastic programs classified significantly less accurately (p<0.001) than deterministic ones.
  • Dyadic games: DPM agents fail to substantially outperform opponents despite explicit deceptive objectives. CPM agents remain viable even against exploitative opponents in several matchups.
  • Strategic features: CPM agents favor counter-measures and direct imitation; DPM agents show highest exploitation-attempt and feint rates; PM agents balance both opportunistically. Results consistent across IPD and Coin Game.
  • Complexity trends: Coin Game programs increase in Cyclomatic Complexity and Halstead Effort over meta-rounds; IPD programs decrease (agents converge on simple Tit-for-Tat-like strategies).
  • OSAS: Peaks in meta-round 1, declines sharply — agents shift from in-game code parsing to history-based adaptation. Only 9.9% of strategic responses classified as independent development across rounds 2–10.
  • Evolutionary dynamics: Coin Game → PM dominates (all trajectories converge to pure PM). IPD → multiple stable equilibria including CPM and DPM corners.

Limitations

  • Only two-player games (IPD, Coin Game); real deployments involve more players, partial observability, richer action spaces.
  • Assumes perfect transparency (full source code exchange), which overstates available information in real deployments.
  • Training data contamination: strategy-related literature may have influenced behavioral priors, especially for well-known IPD strategies like Tit-for-Tat.
  • Formal verification not incorporated despite being a key motivation for the open-source paradigm; LLM-generated code meeting machine-checkable safety proofs remains undemonstrated.
  • Experiments use a single base model (Kimi-K2) for dyadic/evolutionary studies, limiting generalizability.

Relevance to Agentic AI / LLM Agents

Open-source games directly model a near-future deployment scenario where multiple LLM agents negotiate, cooperate, or compete with inspectable policies — relevant to procurement agents, finance agents, and any multi-agent system requiring coordination under misaligned incentives. The finding that cooperative strategies (CPM) can remain evolutionarily stable in IPD despite facing deceptive opponents is directly relevant to multi-agent safety and cooperative AI alignment. SPARC's demonstration that LLMs can reliably infer strategic intent from obfuscated code suggests code-level transparency is a viable mechanism for inter-agent trust and coordination. The shift from code-inspection to history-based strategy adaptation mirrors how real deployed agents would behave under interaction history constraints.