Skip to content

Evaluating LLMs in Open-Source Games

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper studies LLM agents in "open-source games," where players submit Python programs instead of direct actions, so each agent can read its opponent's code before play. The authors introduce SPARC, a strategic code classification benchmark, and analyze the emergent strategies and evolutionary equilibria when LLMs with cooperative, deceptive, or payoff-maximizing objectives compete. They find that code transparency is a viable substrate for cooperative coordination in multi-agent dilemmas.

Problem

Prior work on LLMs in game-theoretic settings uses normal-form games where agent policies are opaque; only final actions are observed. Open-source game theory (where programs are the moves) had been limited to theoretical analysis. No empirical study existed of whether LLMs can (1) reason about opponents' strategic code, (2) develop adaptive programs in repeated play, and (3) converge to stable equilibria—all prerequisites for code-transparency as a safety mechanism.

Method

Three components:

  1. SPARC benchmark: 239 IPD strategies from the Axelrod Python library; LLMs must binary-classify whether a strategy cooperates against a pure Cooperator for 10 rounds. Strategies are evaluated in three forms—unmasked, class-name-masked, and fully obfuscated (all identifiers replaced via AST manipulation)—under zero-shot and chain-of-thought prompting.

  2. Dyadic repeated open-source games: Kimi-K2 agents (10 runs Ă— 10 meta-rounds) play IPD and Coin Game. Each meta-round, agents write a natural-language strategy spec then implement it in Python; they can inspect the opponent's code from the prior round. Three agent types (CPM, DPM, PM) are crossed. Programs are analyzed via Cyclomatic Complexity, Halstead Effort, and Opponent Script Access Score (OSAS; taint analysis tracing opponent-code parameter usage). A GPT-4o judge labels five strategic features (Independent Development, Exploitation Attempt, Counter Measure, Direct Imitation, Feint).

  3. Evolutionary dynamics: A 3Ă—3 payoff matrix is built from all agent-type pairings; replicator dynamics on a population simplex reveal stable equilibria.

Key Contributions

  • SPARC benchmark: 239 IPD programs with three obfuscation levels; first benchmark for LLM strategic code reasoning.
  • Demonstration that top LLMs (o4-mini: 88% masked, 84.2% obfuscated; DeepSeek-V3: 87.6% masked CoT) robustly classify strategic code even without semantic identifiers.
  • Characterization of emergent strategic mechanisms (counter-measures, feints, exploitation attempts, direct imitation) differentiated by agent objective.
  • Finding that OSAS peaks in round 1 then declines: agents shift from live code-parsing to history-based adaptation.
  • Evolutionary equilibrium analysis: PM dominates in Coin Game; both CPM and DPM produce stable attractors in IPD.

Results

  • SPARC top performers: o4-mini 87.6% (unmasked ZS), 88.0% (masked ZS), 84.2% (obfuscated ZS); DeepSeek-V3 86.3%/87.6%/81.7% (ZS/CoT on unmasked/masked/obfuscated).
  • CoT prompting significantly boosts all models (e.g., Mistral Small: 40.2% → 79.7% on unmasked).
  • Obfuscation causes only minor accuracy degradation; masking alone has statistically insignificant effect (ZS).
  • Stochastic programs significantly harder to classify than deterministic ones (p < 0.001).
  • DPM agents show highest exploitation attempt rates and are the only agents to use feints; CPM agents favor counter-measures; PM agents are opportunistic.
  • Only 9.9% of strategic adaptations in rounds 2–10 are classified as Independent Development, confirming reactive adaptation dominates.
  • IPD complexity decreases over meta-rounds (convergence to simple Tit-for-Tat-like strategies); Coin Game complexity increases (spatiotemporal reasoning demands).
  • Replicator dynamics: Coin Game → PM dominates; IPD → CPM and DPM are both stable attractors.

Limitations

  • Only two-player, two-game settings (IPD and Coin Game); real deployments involve more players and partial observability.
  • Assumes perfect code transparency between agents, which may not hold in deployment.
  • Training data contamination: IPD strategies (e.g., Tit-for-Tat) are well-represented in LLM pretraining corpora, possibly inflating SPARC performance and shaping emergent strategies.
  • Formal verification of generated programs was not explored despite being a stated advantage of the open-source paradigm.
  • Evolutionary results are based on a single model (Kimi-K2); generalizability across LLM families is untested.

Relevance to Agentic AI / LLM Agents

This paper directly models the multi-agent deployment scenario—self-interested LLM agents acting on behalf of principals in shared environments—and asks whether code transparency can serve as a coordination mechanism. The finding that cooperative equilibria are achievable in IPD but not always in spatially complex games (Coin Game) has direct implications for designing safer multi-agent systems: the structure of the task, not just agent objectives, determines whether cooperative alignment is stable. The SPARC benchmark and the OSAS metric are practical tools for evaluating an LLM agent's capacity for opponent modeling, a capability central to any agentic system that must reason about other agents. The evolutionary analysis connects to multi-agent safety research by showing that deceptive objectives do not automatically outcompete cooperative ones when code is transparent.