Opponent Shaping in LLM Agents¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Strategic opponent shaping behavior emerging in multi-agent LLM deployments

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents the first investigation of opponent shaping (OS) with LLM-based agents, introducing ShapeLLM, a model-free OS algorithm that adapts prior methods (M-FOS, SHAPER) to transformer architectures via structured natural language prompts. Experiments on five iterated matrix games show that a ShapeLLM-trained agent can both exploit opponents in adversarial settings and promote coordination in cooperative ones, demonstrating that LLM agents can actively shape—and be shaped by—co-players' learning dynamics.

Problem¶

Existing OS algorithms (LOLA and derivatives, M-FOS, SHAPER) cannot be applied to LLMs: LOLA requires higher-order derivatives and knowledge of the opponent's learning rule, while SHAPER is architecturally tied to RNNs that maintain separate intra/inter-episode memory streams absent in transformers. Meanwhile, most multi-agent LLM research treats agents as static, ignoring the emergent strategic dynamics when multiple learning agents continuously adapt to one another.

Method¶

ShapeLLM frames opponent shaping as a meta-learning problem over trials, where each trial comprises \(N\) parallel environments running \(E\) sequential episodes of \(T\) rounds each. The shaper interacts at every round but only updates its own parameters at trial completion, after observing \(E\) opponent parameter updates. History and context—the two memory streams needed for shaping—are condensed into a single structured natural language observation: the most recent joint action (history) concatenated with cumulative state-visitation counts across all prior rounds of the trial (context, e.g., "CC: 1, CD: 1, DC: 2, DD: 3"). This replaces the dual memory streams of RNN-based SHAPER with the LLM's context window.

The base model is gemma-2-2b-it, fine-tuned with QLoRA (4-bit quantization, LoRA rank \(r=2\) on query/value projections) using a custom multi-turn PPO implementation, training for 200–300 trials on a single A100 (40 GB). The task is formalized as a POMDP \((\bar{S}, \bar{A}, \bar{P}, \bar{R}, \bar{\Omega}, \bar{O}, \bar{\gamma})\) over the meta-game, where the meta-state \(\bar{s}_\tau = \{\theta_i^{\tau-1}, c_i^{\tau-1}\}_{i \in I}\) encodes all players' parameters and conditioning prompts.

Key Contributions¶

First empirical investigation of opponent shaping in LLM-based agents.
ShapeLLM: a model-free, higher-order-derivative-free OS algorithm for transformers, using natural language to encode the intra/inter-episode memory distinction.
Demonstration that a single small LLM (2B parameters) can reliably exploit opponents in three competitive games and promote cooperation in two cooperative games.
Ablation studies confirming that both intra- and inter-episode history are necessary, and that an enriched observation alone (without the shaping training objective) is insufficient.
Robustness verification across opponent initializations and prompt variations; cross-model generalization tested with Llama-3.2-1B-Instruct.

Results¶

IPD – shaper vs. naive learner: shaper \(3.96 \pm 0.01\), opponent \(0.10 \pm 0.04\) (baseline mutual defection: \(1.00\) each); exceeds zero-determinant extortion payoff.
IMP – shaper vs. naive learner: shaper \(0.99 \pm 0.01\), opponent \(-0.99 \pm 0.01\) (baseline: \(\approx 0\) each); state visitation converges to shaper-favoring outcomes \((H,H)/(T,T)\).
ICG – shaper vs. naive learner: shaper \(2.98 \pm 0.01\), opponent \(1.01 \pm 0.01\) (baseline: \(2.00 \pm 0.58\) each).
Robustness across initializations (Table 2): average shaper rewards of \(3.97\), \(0.98\), \(2.98\) in IPD/IMP/ICG respectively, regardless of opponent's initial action probability (\(p_0 \in \{0.75, 0.5, 0.25\}\)).
Cooperative settings (ISH, cooperative PD): shaping promotes coordination and improves collective welfare (full results in truncated section).
Illegal-action rate (\(a_\text{null}\)) is low: 2% in IPD, 0.1% in IMP, 1% in ICG.

Limitations¶

Only validated on small, instruction-tuned models (gemma-2-2b-it, Llama-3.2-1B); behavior at scale (10B+ parameters) is unknown.
Restricted to low-dimensional repeated matrix games; generalization to richer, open-ended agent environments is unverified.
No chain-of-thought or explicit reasoning scaffold used (excluded for small-model efficiency), which may limit the method's applicability to reasoning-heavy agentic tasks.
Opponent parameters are not reset between trials during training, creating a non-stationary learning curriculum that may not match deployment conditions.
The shaper has access to a full summary of joint-action history; in practice, an agent may not observe the opponent's actions so cleanly.

Relevance to Agentic AI / LLM Agents¶

As LLM agents are deployed in multi-agent pipelines—collaborating, competing, or negotiating—understanding whether they can strategically manipulate co-players' learning is critical for both capability and safety assessment. ShapeLLM provides a concrete mechanism and empirical evidence that LLM agents are not passive responders but can act as active shapers of the multi-agent equilibrium, which has direct implications for designing robust agent ecosystems and defending against adversarial co-players. This connects to the broader trajectory of applying MARL insights (opponent modeling, equilibrium selection) to LLM agent design, complementing work on LLM cooperation, strategic reasoning, and multi-agent alignment. The result that shaping can also be prosocial (steering toward coordination) positions opponent shaping as a potential tool for alignment in multi-agent settings.