Skip to content

Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios

🕒 Published (v1): 2025-11-30 20:25 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a three-agent framework for simulating human users in goal-oriented conversations, decomposing simulation into orchestration, task-state tracking, and behavioral-attribute generation. Tested on 60 restaurant-ordering scenarios, the full system achieves a 102.6% improvement in composite simulation quality (CRRS: 0.818) over a single-LLM baseline (0.404). The architecture makes agent decision traces fully auditable via tool-call logging, yielding near-perfect Decision Explainability Index (0.994).

Problem

Static test sets and human evaluators cannot scale to diverse, multi-turn conversational AI testing. Single-LLM user simulators conflate state tracking, persona maintenance, and response generation into one context, producing inconsistent personas, poor task completion, and opaque decision traces.

Method

A three-agent pipeline coordinated through strict tool-call sequencing:

  1. User Agent (orchestrator): generates \(r_t = f_\text{user}(m_t, s_t, a_t)\) by invoking the two sub-agents before each response.
  2. State Tracking Agent: maintains \(s_t = \{T_\text{current}, T_\text{target}\}\); updates are monotonic (add/remove only, never implicit modify); exit condition is \(T_\text{current} \supseteq T_\text{target}\).
  3. Message Attributes Generation Agent: outputs \(a_t = \{\)mood_tone, task_execution_style, exploration_style, task_completion_status\(\}\) conditioned on persona biography \(p_\text{bio}\) and \(s_t\).

Tool calls are strictly sequenced (State Tracking → Message Attributes Generation); structured Pydantic AI models enforce type-safe data contracts. Persona consistency is evaluated via Persona Adherence Score (PAS), behavioral naturalness via Behavioral Variance Score (BVS, peaks at 20% transition rate), task accuracy via TRA (F1 over normalized item sets), and explainability via DEI (tool-invocation density normalized to message count).

Key Contributions

  • Three-agent architecture separating working memory (state tracking), behavioral planning (attribute generation), and language production (user agent orchestration).
  • Five quantitative metrics — PAS, BVS, TRA, DEI, CRRS — with explicit formulas designed for conversational user simulation.
  • Systematic ablation across five configurations isolating each component's contribution.
  • 60-case restaurant-ordering dataset pairing 20 personas × 3 order complexities.

Results

  • CRRS: Config5 (full) 0.818 vs. Config1 (single LLM) 0.404 → +102.6% (\(p < 0.001\)).
  • BVS: 0.839 vs. 0.218 baseline → +284.5% (\(p < 0.001\)); Config4 (MAG only, no state tracking) collapses to BVS = 0.000.
  • TRA: Config3 and Config5 both reach 0.785 vs. 0.608 baseline → +29.1% (\(p < 0.01\)); State Tracking Agent is the sole driver of TRA gains.
  • PAS: 0.706 vs. 0.589 → +19.9% (\(p < 0.01\)).
  • DEI: 0.994 (Config5) vs. 0.000 (Config1).
  • Cost: full system uses 14,789 avg. tokens and 23.16 s latency vs. 6,618 tokens / 5.08 s for baseline (~124% more tokens, ~356% more latency).

Limitations

  • Substantial compute overhead: ~124% more tokens and ~356% higher latency than single-LLM baseline.
  • Each new domain requires bespoke prompt engineering and state schema design.
  • Behavioral model omits complex human dynamics: indecision, social norms, cultural nuance.
  • Validation is English-only, single-domain, 60 test cases — limited generalizability.
  • No cross-lingual or real-world deployment evaluation.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instance of a testing harness built from a multi-agent meta-harness pattern: a coordinator agent invokes specialized sub-agents in a fixed protocol to produce structured, reproducible outputs — the same architectural blueprint used in paper-digest and evaluation harnesses. The strict tool-call sequencing, monotonic state constraints, and composite scoring metrics directly parallel quality-gate patterns in automated pipeline harnesses. The ablation methodology (isolating each harness component's marginal contribution) provides a reusable evaluation template for anyone building or auditing multi-agent orchestration pipelines. The DEI metric — measuring decision traceability via tool-invocation density — is directly applicable as a health metric for harness observability.