Step-Level Preference Learning for Generative Agents in Social Simulations

🕒 Published (v1): 2026-07-16 01:58 UTC · Source: Arxiv · link

Why this paper was selected

Step-level preference learning for long-horizon agent decisions; advances beyond episode-level RLHF for agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimPref introduces step-level human preference supervision for generative agent (GA-style) social simulations, collecting 57K annotated preference pairs across six decision modules (planning, memory importance, reflection, action, dialogue). Fine-tuning open-weight LLMs with SFT and DPO on this data consistently improves simulation fidelity, coordination, and goal fulfillment across held-out social events, narrowing the gap to proprietary models and human reference behavior.

Problem

Existing GA-style social simulation agents receive only trajectory-level or Likert-scale feedback collected after simulation ends, providing no signal about which intermediate decisions (planning, memory importance scoring, reflection, action selection) caused downstream failures. This granularity gap prevents attribution of failures to specific module outputs and limits the quality of training signal for improving long-horizon social behavior.

Method

The paper presents a three-stage framework:

1. SimPref interface: Annotators control 2–3 agents during live simulation. At each module trigger, GPT-4o generates three candidate outputs from the current decision context \((o_{i,t}, g_i, R_{i,t}, z_{i,t})\); the annotator selects the preferred output or writes a custom alternative. Rejected candidates and the accepted output form pairwise preference tuples \((x^{(k)}_{i,t}, y^+, y^-)\).

2. Dataset construction: 57,239 preference pairs spanning 30 social events (4–8 agents each, partial observability, asymmetric information distribution) and six modules: Plan, Act, Talk, Reflect, Reflection Question, Importance Score.

3. Preference learning: Open-weight base models (Qwen2.5-7B/14B, Llama-3.1-8B) are first fine-tuned via SFT on accepted outputs \(y^+\) conditioned on \((x, k)\), then further aligned via DPO on \((x, y^+, y^-)\) pairs warm-starting from SFT checkpoints.

Key Contributions

• SimPref, the first interactive interface enabling real-time step-level human supervision of GA-style simulation module outputs.
• First step-level human preference dataset for GA-style social simulation: 57,239 preference pairs across 6 modules and 30 social events.
• Empirical demonstration that step-level SFT+DPO substantially improves open-weight LLMs on 10 held-out events across five trajectory-level metrics.
• Behavioral distribution analysis showing KL divergence to human reference drops from 0.610 to 0.084 after training, with event coordination time rising from 8% to 36%.
• Ablation isolating memory importance estimator (\(+\)MI) showing that retrieval-only improvement is insufficient; coordinated multi-module alignment is necessary.

Results

• SFT alone produces the majority of gains; e.g., Q7B: Location Adherence +0.76, Temporal Adherence +1.05, Requirement Consistency +1.14 (all absolute on a 5-point scale).
• SFT+DPO adds modest further improvements concentrated in Location Adherence (+0.21 over SFT averaged across three models) and Interaction Quality (+0.08); slightly negative on Temporal Adherence (−0.04).
• L8B-SFT+DPO reaches Role Fulfillment 3.08, matching or approaching GPT-5.2 (3.05) and DeepSeek-v3.2 (3.02) on that metric.
• Human–LLM Top-1 agreement ranges from 29.3% (Importance Score, below random 33% baseline) to 51.4% (Reflection Question), revealing systematic miscalibration of memory importance scoring.
• \(+\)MI-only variants show small and inconsistent gains (e.g., DeepSeek-v3.2+MI: +0.13 Location, −0.17 Temporal), confirming retrieval alone is insufficient.
• Behavioral KL divergence to human reference: 0.610 (base) → 0.084 (SFT+DPO).
• Misleading interactions reduced from 8% to 2%; idle/off-task from 31% to 22%.

Limitations

• Annotations are single-annotator per step with no inter-annotator reliability reported, raising concerns about label noise and consistency.
• Evaluation relies on LLM judges (GPT-5.2, DeepSeek-v3.2) using 5-point Likert scores, which may not capture all dimensions of social realism.
• Dataset collected from 30 training events designed by the same team; generalization to naturalistic or open-ended social settings is unverified.
• DPO gains are modest and dimensionally uneven; the mechanism by which DPO helps (contrastive discrimination vs. calibration) is not fully characterized.
• Scenarios are constrained multi-agent events with explicit goals; behavior in open-ended sandbox simulations remains untested.
• Annotation was performed by eight software engineers, not domain-diverse crowd workers, potentially limiting preference diversity.

Relevance to Agentic AI / LLM Agents

This work directly addresses a core challenge in LLM agent alignment: the absence of fine-grained training signal for intermediate reasoning and decision steps within a multi-module architecture. By decomposing agent execution into supervisable step-level decisions and applying DPO at that granularity, it provides a concrete methodology for aligning agentic pipelines where trajectory-level feedback is too coarse — a paradigm directly applicable to tool-calling agents, planning agents, and multi-agent coordination systems. The finding that memory importance scoring is systematically miscalibrated (human–LLM agreement below random) is a notable diagnostic for retrieval-augmented agent architectures broadly. The KL-to-human alignment result (0.610 → 0.084) demonstrates that local step-level supervision translates into coherent global behavioral change, validating step-level preference learning as a practical alternative to costly trajectory-level RLHF in agentic settings.