Skip to content

Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MaKTO (Multi-agent Kahneman-Tversky's Optimization) fine-tunes LLMs for strategic social deduction games by integrating decision-making and language expression into a single model trained via offline preference optimization. Using Werewolf as a testbed, it achieves 61% average win rate against diverse LLM/RL baselines and 60% against expert human players, with Turing-test detectability below chance (48.9%).

Problem

Prior Werewolf AI systems decouple decision-making from language generation (RL for intent, LLM for expression), which compresses the language strategy space, limits generalization, and violates the intrinsic unity of language and action posited by Wittgenstein's language game theory. Win/loss trajectory rewards also fail to identify which individual actions within a game were actually optimal.

Method

MaKTO is a two-stage training pipeline:

  1. Behavior cloning (SFT): Fine-tunes a base LLM (Qwen2.5-14b/72b-instruct) on a purpose-collected expert dataset of 331 annotated Werewolf games (17 expert players, 1000+ games each), structured into a "think-before-respond" chain-of-thought format covering game terminology, advanced strategies, and annotated action rationales.

  2. Multi-agent KTO: Generates unpaired desirable/unacceptable responses by deploying the target model against a diverse pool (SFT variants, GPT-4o, Claude-3.5, RL+LLM Mix agents). Stepwise preference labels—rather than win/loss trajectory labels—are assigned via three methods:

  3. Heuristic-based: role-specific rules (e.g., werewolf kills special roles = desirable).
  4. Staged voting-based: players voted out by the majority gave suboptimal speeches.
  5. Verifier-based: GPT-4o checks factual consistency and logical coherence of speeches.

KTO (not DPO) is chosen because it handles unpaired desirable/unacceptable samples, suits sparse multi-agent trajectories, and converges more reliably than online RL in large dialogue action spaces.

Key Contributions

  • MaKTO algorithm: Multi-agent KTO training that jointly optimizes language expression and strategic decision-making without requiring paired preference data.
  • Expert Werewolf dataset: 331 annotated games from 17 expert players with full chain-of-thought reasoning for actions, speeches, and votes; 51 games held out for evaluation.
  • Stepwise preference selection: Three complementary labeling methods (heuristic, voting-based, verifier-based) that identify quality at the action level rather than trajectory level.
  • Comprehensive evaluation: Tournaments vs. LLM/RL baselines, head-to-head against expert humans, Turing-style blind detectability tests, and cross-setting generalization experiments.

Results

  • Inter-agent tournament (9-player Seer-Witch-Guard): MaKTO-72b achieves 61.0% average win rate vs. 49.6% for GPT-4o (+23.0% relative) and 55.0% for Mix agent (+10.9% relative).
  • Human competition: 60% win rate (5/10 as villagers, 7/10 as werewolves) against expert players (1000+ games each).
  • Turing-test detectability: 48.9% detection accuracy for MaKTO-72b vs. 76.6% for GPT-4o — below random chance.
  • Random competition (30 games, mixed human+AI): MaKTO-72b 61.8% ± 8.3%, ranking 4th among all players, above average human win rate of 54%.
  • vs. DeepSeek-R1-Distill: MaKTO-14b beats R1-Distill-Qwen-14b 74%; MaKTO-72b beats R1-Distill-Llama-70b 84%.
  • Generalization to new roles/scales: MaKTO-72b achieves highest win rate in unseen 10-player (Guard: 0.625, Hunter: 0.585) and 12-player (0.605) settings; Mix agent collapses when Guard is replaced by Hunter.
  • Ablation: Multi-agent play beats self-play KTO by 4%; stepwise selection beats trajectory-based by ~8.7%; SFT alone provides +30.6% over base, KTO refinement adds +8.7%.

Limitations

  • Experiments fix system prompts without role-playing personas, potentially underestimating LLM expressiveness variation.
  • Turn-based conversation structure only; free-form simultaneous multi-agent communication is not modeled.
  • Residual hallucination and inconsistency in long conversations; long-context coherence remains a weakness.
  • Online multi-agent RL may outperform offline KTO but is not explored here; long-CoT integration is also left as future work.
  • Dataset and evaluation are restricted to Werewolf; transfer to other SDGs or open-domain multi-agent tasks is unverified.

Relevance to Agentic AI / LLM Agents

MaKTO directly addresses a core challenge in multi-agent LLM systems: how to train agents that integrate language and strategic action without decoupling them into brittle pipeline stages. The use of KTO with unpaired, stepwise preferences provides a practical offline RL recipe for environments where paired preference data is unavailable — a common constraint in real agentic deployments. The multi-agent diverse-pool training strategy (rather than self-play) is a transferable insight for any system where agents must generalize across heterogeneous opponents or collaborators. The near-human Turing-test result also benchmarks the frontier of socially-embedded LLM agent behavior.