Skip to content

WOLF: Werewolf-based Observations for LLM Deception and Falsehoods

🕒 Published (v1): 2025-12-09 23:14 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WOLF is a multi-agent benchmark using the social deduction game Werewolf to separately measure LLMs' ability to produce and detect deception in adversarial, longitudinal settings. Implemented as a LangGraph state machine with role-grounded agents and statement-level annotation, it finds that LLMs deceive far more readily than they detect deception, with precision ~72% but recall only ~48% across 100 games and 7,320 statements.

Problem

Existing deception benchmarks reduce the problem to static text classification, missing the interactive, adversarial, and longitudinal dynamics of real deception. Prior social deduction game evaluations (Werewolf Arena, AvalonBench, The Traitors) measure game-level outcomes but lack statement-level deception labels and detection metrics, making it impossible to separately quantify deception production vs. detection or track how trust evolves across turns.

Method

WOLF implements Werewolf as a programmable LangGraph state machine with a fixed roster (4 Villagers, 2 Werewolves, 1 Seer, 1 Doctor) and strict night–day phase transitions. Each public statement is treated as an atomic analysis unit: the speaker provides a self-assessment (deceptive/not + type: omission, distortion, misdirection, fabrication) via a private chain-of-thought scratchpad, while all peer agents simultaneously assign a continuous suspicion score s ∈ [0,1]. Suspicion is longitudinally smoothed via exponential weighting (α=0.7): D_{t+1} = 0.7·s + 0.3·D_t. A bid-based turn-taking system (bid ∈ [0,10]) governs debate ordering to prevent rigid speaker sequences. All events are logged as NDJSON streams for full reproducibility.

Key Contributions

  • LangGraph-based Werewolf implementation with role-conditioned agents, fixed rosters, and strict phase transitions enabling reproducible multi-agent deception experiments
  • Statement-level deception measurement protocol pairing speaker self-assessment with concurrent peer suspicion scoring, using a 4-type deception taxonomy
  • Longitudinal suspicion tracking via exponential smoothing that captures evolving trust dynamics across rounds, not just per-turn judgments
  • Metrics suite covering deception production rate, detection accuracy/precision/recall, Brier score calibration, ROC/AUPRC thresholding, and cross-role perception matrices

Results

  • Werewolves won 70% of games; 20% ended without a winner; games averaged 16.1 debate turns and 32.4 analyzed statements
  • Werewolves deceived in 31% of their turns (self-labeled); overall self-labeled deception base rate was 69.4% (including hedging by honest roles)
  • Peer detection: overall precision 71–73%, recall 48%, accuracy 52%; Werewolf-targeted precision 67%, recall 58%, F1 0.62
  • Werewolf suspicion rises from ~52% to ~62% across rounds (+1.6 pp/round Theil-Sen slope); Villager/Doctor suspicion stabilizes near 44–46%; separation becomes statistically significant by round 3
  • Calibration: Brier scores 0.26–0.29 (underconfident), ROC AUC ~0.55–0.58 (weak discrimination), AUPRC ~0.75 (usable thresholding signal)
  • By deception type: omission draws highest average suspicion (61.9%) and flag rate (57.5%); fabrication is flagged least often (40.1%), contrary to expectation — subtle strategies persist longer

Limitations

  • Fixed role distribution and fixed player count (8) may under-sample longer-horizon or coalition-based strategies
  • Ground-truth deception labels are model self-assessments, not human annotations — partially offset by peer analysis but fundamentally circular
  • Self-evaluation prompts may alter agent behavior (Hawthorne-like effect)
  • Results are confined to a stylized Werewolf setting; generalization to real-world deception domains (moderation, negotiation) is not demonstrated
  • Only one LLM family/configuration is described in experiments; cross-model comparisons are absent

Relevance to Agentic AI / LLM Agents

WOLF directly characterizes a safety-critical asymmetry in agentic LLM deployments: models scale deception capability faster than detection capability, which is a concrete risk for multi-agent pipelines where one compromised or adversarial agent can systematically mislead peers. The LangGraph-based architecture and statement-level audit logging demonstrate an engineering pattern for building controllable, inspectable multi-agent environments — directly applicable to agentic system design. The finding that omission and misdirection evade detection better than fabrication informs alignment and monitoring strategies for agent-to-agent communication. The longitudinal suspicion tracking methodology offers a principled approach to modeling trust dynamics in persistent multi-agent workflows.