Skip to content

LLM-PySC2: Starcraft II learning environment for Large Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM-PySC2 is an open StarCraft II learning environment that exposes the full PySC2 action space (~500 raw actions, 100+ text actions for Protoss) to LLMs with multi-modal observations and a native multi-agent framework. It is the first such environment to support complete pysc2 action coverage and asynchronous multi-agent querying. Evaluations across 9 mainstream LLMs reveal persistent failure modes—hallucinations, domain ignorance, and inefficient coordination—showing SC2 remains unsolved for LLMs.

Problem

Existing LLM-SC2 interfaces (SwarmBrain, TextStarCraft II) drastically truncate the action space and reduce continuous actions to discrete ones, masking the true difficulty of the game. Simultaneously, no prior platform provides native multi-agent (MA) collaboration for LLMs in SC2, nor does any expose multi-modal observations (RGB screen, minimap, feature maps) alongside structured game Wiki knowledge. The vector interfaces of vanilla SC2LE are incompatible with LLM text I/O.

Method

LLM-PySC2 wraps SC2LE at the player level. Text actions follow a <ActionName(args)> syntax; a bridge-object protocol parses them into pysc2 function calls (e.g., <Select_Unit_Blink_Screen(tag, screen)> → select_point + Effect_Blink_screen). Observations are assembled per-agent by an observation wrapper that produces: (1) structured text (global info, unit status, valid actions, communication log, Wiki snippets) and (2) four image streams (RGB screen, RGB minimap, RGB feature, original feature maps) encoded as base64 and appended to OpenAI-format query messages. Agents run in independent threads querying remote LLM servers; an asynchronous architecture keeps per-step LLM latency constant regardless of agent count. The MA framework defines four hierarchical agent roles—Commander, Developer, Builder, CombatGroup—communicating via <MessageTo(target, content)> actions in natural language. Two "Easy Modes" (EasyControl, EasyBuild) allow researchers to isolate specific agent layers.

Key Contributions

  • First SC2 environment exposing the complete pysc2 action space to LLMs via a structured text-action protocol.
  • Native multi-agent framework supporting centralized and decentralized decisions with asynchronous LLM querying (constant latency regardless of agent population).
  • Multi-modal observation interface: RGB screen, minimap, feature maps, text state, and structured Wiki knowledge per agent.
  • Eight new micro-operation scenarios (LLM-PySC2 task group) requiring unit-skill usage and task planning, beyond classic SMAC.
  • Empirical evaluation of 9 LLMs (GPT-3.5-turbo through GPT-o1-mini, LLaMA 3.1 8B–405B, GLM-4-plus, Claude 3 Haiku) across macro- and micro-decision settings with KD ratio and Winning Rate metrics.

Results

  • Macro-decisions (ECEB mode, GPT-4o-mini, 30 runs): 100% WR at L1, ~57% at L4, ~0% at L6–L7.
  • SCEB (full action space, easy build): 0% WR from L3 onward due to micro-operation errors.
  • ECSB (easy control, standard build): 60% WR at L3, 17% at L4, 0% at L6+.
  • LLM-SMAC: All tested LLMs achieve 0% WR on most scenarios (best: GPT-4o 30% on 1c3s5z); KD ratios consistently <1.
  • LLM-PySC2 task group (task3, intercept airdrops): LLaMA-405b achieves 100% WR; GPT-4o achieves 100% WR with infinite KD; GPT-4o-mini achieves 0% on same task.
  • Task4 (Blink-Stalker): All models 0% WR across the board.
  • Latency: Environment-side processing <0.1s/step; LLM inference dominates at 5.2–5.6s/query (GPT-3.5).
  • Scaling law finding: LLaMA-405b does not substantially outperform LLaMA-70b on most tasks; GPT-o1-mini (reasoning model) shows no significant advantage over standard models in novel environments.

Limitations

  • Only Protoss faction fully detailed; Terran and Zerg action spaces exist in SC2LE but are not explicitly benchmarked.
  • No fine-tuning or RAG-augmentation baselines; all models tested zero-shot/few-shot with prompt engineering only.
  • Small sample sizes (20–30 runs per setting) necessitate Wilson Score CIs but statistical power remains limited.
  • Real-time constraint not met: 5+ second LLM latency per step precludes competitive play against built-in bots above level 5.
  • MA coordination bugs (unconditional teammate trust, noisy communication) are identified but not addressed by the environment itself.
  • Complete game mode (SCSB, full action + full build) was not evaluated due to poor LLM performance at lower modes.

Relevance to Agentic AI / LLM Agents

LLM-PySC2 provides one of the most demanding open benchmarks for LLM-based agentic decision-making: hierarchical multi-agent coordination, long-horizon planning, real-time action selection over a 100+ action space, and multi-modal perception—all simultaneously. The empirical finding that neither chain-of-thought reasoning (o1-mini) nor parameter scaling (405B vs. 70B) reliably closes the gap directly challenges assumptions common in LLM agent work, pointing toward domain-specific training or retrieval as necessary complements. The asynchronous multi-agent architecture—where agents communicate via natural language and query independent LLM backends—serves as a concrete reference implementation for distributed LLM agent systems at non-trivial scale.