Skip to content

Improving General Role-Playing Agents via Psychology-Grounded Reasoning and Role-Aware Policy Optimization

🕒 Published (v1): 2026-06-25 13:34 UTC · Source: Arxiv · link

Why this paper was selected

Psychology-grounded reasoning + role-aware policy optimization; moves beyond SFT mimicry for general-purpose persona agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RAPO introduces two components for general-purpose role-playing agents: Psy-CoT, a psychology-grounded chain-of-thought decomposing pre-response reasoning into Interaction Perception, Psychological Empathy, and Logical Construction; and Role-Aware Policy Optimization (RAPO), which weights RL gradients asymmetrically by profile–token mutual information to suppress reward hacking from role-agnostic generic phrases. Together they achieve consistent improvements over GRPO and SFT baselines across three benchmarks.

Problem

SFT-based role-playing agents memorize surface behavioral patterns without genuine character understanding, failing to generalize to out-of-distribution roles. Generic CoT methods push models toward objective reasoning rather than subjective, character-idiosyncratic reasoning. Standard RL (e.g., GRPO) assigns equal gradient signals to role-specific and role-agnostic tokens: reward-hacking generic phrases that satisfy the LLM-judge accumulate identical learning signal as genuinely character-faithful expressions, causing the model to treat them as equally optimal over training.

Method

Psy-CoT structures pre-response reasoning into three sequential steps before producing the in-character answer: 1. Interaction Perception (Theory of Mind): parse the scene, infer the interlocutor's intentions and mental state. 2. Psychological Empathy (cognitive appraisal + emotion regulation theory): filter the situation through the character's background and values, identify the genuine emotional reaction, decide how much to express/mask/weaponize. 3. Logical Construction (goal-directed behavior theory): synthesize facts through the character's worldview and select a short-term objective the character would naturally pursue, including impulsive or suboptimal ones.

RAPO modifies the GRPO objective with per-token gradient weighting derived from profile–token mutual information: $\(I_t = I(y_t; P \mid x) = H(y_t \mid x) - H(y_t \mid P, x)\)$ where \(H(y_t \mid P, x)\) is computed with the role profile \(P\) and \(H(y_t \mid x)\) without it. Tokens with \(|I_t| < \delta\) receive weight \(w_t = 1\). For role-specific tokens (\(|I_t| \geq \delta\)), weighting is asymmetric: $\(w_t = \begin{cases} 1 + \text{clamp}(|I_t| \cdot \lambda,\, 0,\, \mu^+) & \text{if } \hat{A} \geq 0 \\ 1 - \text{clamp}(|I_t| \cdot \lambda,\, 0,\, \mu^-) & \text{if } \hat{A} < 0 \end{cases}\)$ Positive advantage amplifies gradients on role-specific tokens; negative advantage attenuates them, preventing the model from associating negative feedback with character-identifying behaviors. The modified objective substitutes these \(w_t\) into the GRPO clipped IS-ratio objective with KL regularization toward the reference model. Rewards decompose into format \(R_\text{fmt}\), think-quality \(R_\text{think}\), and answer-quality \(R_\text{ans}\), the latter two evaluated by an LLM judge with rubrics aligned to the Psy-CoT steps. Training uses 25k instances from HER and 5k from AdaRPSet-Synthesis; each sample is stored with a profile-ablated version to enable efficient paired forward passes for \(I_t\) computation.

Key Contributions

  • Psy-CoT: psychology-grounded CoT framework with three role-specific reasoning steps (Interaction Perception, Psychological Empathy, Logical Construction), enabling subjective/idiosyncratic rather than generic deliberation.
  • RAPO: per-token gradient modulation via profile–token mutual information, asymmetrically amplifying role-specific signals under positive advantage and attenuating penalties under negative advantage to prevent reward hacking.
  • Structured reward decomposition (format + think + answer) with LLM-as-a-judge aligned to Psy-CoT rubrics.
  • Demonstrated cross-lingual generalization to CharacterEval (Chinese) and cross-architecture generalization across Qwen2.5-7B, Qwen3-4B, and Llama-3.1-8B.

Results

  • Psy-CoT on Llama-3.3-70B-Instruct: +5.2% average on CoSER, +4.2% on CharacterBench vs. vanilla (no CoT); outperforms CB-CoT on both benchmarks.
  • RAPO vs. GRPO on Qwen2.5-7B (CoSER): avg 47.74 vs. 45.82; Anthropomorphism +2.15; Believability on CharacterBench +0.10; Behavior sub-metric on CharacterEval 3.81 vs. 2.74 (+39%).
  • RAPO vs. untrained baseline (CoSER): Qwen2.5-7B +13.7% (41.31→47.74), Qwen3-4B +15.6% (42.11→47.87), Llama-3.1-8B +40.1% (31.69→44.55).
  • RAPO-trained Qwen3-4B (47.87) and Qwen2.5-7B (47.74) surpass GPT-5-Chat (45.88) and DeepSeek-V4-Flash (45.49) on CoSER.
  • RAPO outperforms CPO, GRPO, GSPO, and DAPO consistently across all three benchmarks.
  • SFT baselines (Crab, CoSER, AdaMARP) overfit to their training distribution; on CharacterBench, Qwen2.5-7B-CoSER (3.50) and Crab (3.62) fall below the base model (3.64); all three SFT variants drop below the untrained baseline on CharacterEval (2.84–2.87 vs. 3.09).

Limitations

  • Psy-CoT degrades smaller models that cannot reliably follow structured CoT instructions (e.g., Qwen2.5-14B drops on CharacterBench from 3.77 to 3.68 with Psy-CoT).
  • Profile–token mutual information requires two forward passes per training sample (with and without profile), increasing training compute.
  • Reward model is LLM-based (Qwen3-30B-A3B-Instruct-2507-FP8), retaining inherent LLM-judge biases; rubric alignment to Psy-CoT steps is author-designed and may not generalize to all character types.
  • Evaluations are limited to three benchmarks; the CoSER benchmark itself uses GPT-4o-mini as judge, creating a potential evaluation–training model entanglement.
  • The method is validated only on encoder-free decoder LLMs; applicability to multimodal or embodied agents is unexamined.

Relevance to Agentic AI / LLM Agents

Role-playing agents are a direct instance of general-purpose LLM agents that must condition behavior on a natural-language specification (the role profile) rather than static fine-tuning — the same challenge faced by instruction-following and tool-use agents that must ground behavior in context-provided constraints. RAPO's profile–token mutual information mechanism is a broadly applicable technique for token-level credit attribution in RL, relevant to any agent trained with GRPO-style objectives where some tokens are policy-relevant and others are generic scaffolding. The reward hacking dynamic RAPO targets — where role-agnostic filler tokens hijack LLM-judge rewards — is a general failure mode in RLHF/RLAIF pipelines for open-ended generative agents, making this work directly informative for reward shaping in coding agents, dialogue agents, and tool-use agents. The Psy-CoT decomposition also offers a structured template for embedding Theory of Mind and goal-directed reasoning into agent deliberation pipelines.