Skip to content

Agentic Persona Control and Task State Tracking for Realistic User Simulation in Interactive Scenarios

🕒 Published (v1): 2025-11-30 20:25 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A three-agent framework (User Agent, State Tracking Agent, Message Attributes Generation Agent) for simulating realistic human users in conversational AI testing, validated on restaurant ordering. The decomposed architecture achieves 102.6% improvement in composite simulation quality over a single-LLM baseline, with statistically significant gains across all metrics.

Problem

Conversational AI systems lack scalable, realistic testing infrastructure. Static test sets miss multi-turn dynamics; human evaluators are expensive and inconsistent. Single-LLM simulators conflate state tracking, persona modeling, and response generation, producing either over-scripted or behaviorally incoherent interactions.

Method

Three specialized agents collaborate through structured tool-call protocols: - User Agent: Orchestrator that invokes the two sub-agents in sequence before generating each response: rt = f_user(mt, st, at). - State Tracking Agent: Maintains {T_current, T_target} (confirmed vs. goal task items); updates are monotonic (add/remove only), ensuring coherence. - Message Attributes Generation Agent: Conditions on persona biography and current state to emit structured behavioral attributes at = {mood_tone, task_execution_style, exploration_style, task_completion_status} with four discrete value sets.

Exit gating fires when T_current ⊇ T_target. Implementation uses PydanticAI with GPT-4o; the ordering system (also GPT-4o) is blind to internal state.

Key Contributions

  • Three-agent decomposition (orchestrator + state tracker + behavior controller) with strict sequencing and monotonic state constraints.
  • Five-metric evaluation suite: Persona Adherence Score (PAS), Behavioral Variance Score (BVS), Task Restriction Adherence (TRA, F1-based), Decision Explainability Index (DEI, tool-call traceability), and Composite Realism and Reliability Score (CRRS).
  • Restaurant-ordering benchmark: 20 personas Ă— 3 order complexities = 60 test cases against an independent LLM ordering system.
  • Ablation study across 5 configurations isolating each component's contribution.

Results

  • Full system (Config5) vs. single-LLM baseline (Config1):
  • CRRS: 0.818 vs. 0.404 (+102.6%, p<0.001)
  • BVS: 0.839 vs. 0.218 (+284.5%, p<0.001)
  • PAS: 0.706 vs. 0.589 (+19.9%, p<0.01)
  • TRA: 0.785 vs. 0.608 (+29.1%, p<0.01)
  • DEI: 0.994 vs. 0.000 (+100%, p<0.001)
  • Config4 (MAG only, no state tracker): BVS = 0.000, showing behavioral control collapses without state grounding.
  • Config3 and Config5 achieve identical TRA (0.785), confirming state tracking alone drives task accuracy.
  • Full system costs ~14,789 tokens/response, 23.16s latency vs. 6,618 tokens/5.08s for baseline.

Limitations

  • 124% token overhead and 356% latency increase over baseline; cost-prohibitive for high-throughput or resource-constrained settings.
  • Domain-specific prompt engineering and state schema required per deployment domain.
  • Behavioral model omits complex human traits: indecision, cultural nuance, social dynamics.
  • Validation is English-only, single domain (restaurant ordering), 60 test cases—severely limited generalization evidence.
  • No comparison to prior user-simulation baselines (different architectural assumptions cited as justification).

Relevance to Agentic AI / LLM Agents

This paper directly instantiates a core agentic design pattern—decomposing a complex cognitive task into specialized agents with explicit memory (state tracker) and behavioral planning (attribute generator) modules, mirroring cognitive architectures like CoALA. For the LLM agent community, it provides a concrete, evaluated blueprint for synthetic user simulation as a scalable testing harness for conversational agents, reducing dependence on human evaluation. The ablation results quantify the synergy penalty of omitting either sub-agent, providing empirical grounding for debates about modular vs. monolithic agent design. The evaluation metrics (especially DEI and CRRS) are reusable for benchmarking other agentic pipelines requiring interpretability alongside task performance.