Skip to content

Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction

🕒 Published (v1): 2025-11-17 14:22 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Mem-PAL introduces PAL-Bench, the first Chinese benchmark for long-term personalized user-agent interaction, paired with H2 Memory—a hierarchical, heterogeneous memory framework that separately encodes behavioral logs and dialogue histories at both concrete and abstract levels to enable preference-aware, context-sensitive responses. The system is evaluated across three tasks: requirement restatement, solution proposal, and multi-turn dialogue interaction.

Problem

Existing dialogue agents treat users uniformly and lack mechanisms to model subjective, evolving user preferences across long-term multi-session interactions. Prior long-term dialogue benchmarks focus narrowly on factual recall from dialogue histories and ignore behavioral logs (app usage, device actions), which are integral to real user-agent scenarios. No public Chinese dataset exists for this setting.

Method

PAL-Set construction: A multi-stage LLM pipeline (Qwen2.5-Max) generates 100 synthetic users, each with a persona (demographics, traits, monthly timeline), session-specific situation entries, diary-style experiences, timestamped behavioral logs, and dialogue frameworks with explicit requirement/solution labels. Human annotators verify and refine the output (avg. scores: 2.75/3 for logs, 2.67/3 for dialogues).

H2 Memory framework: Four memory components organized in two tiers: - Concrete level: (1) Log Graph (MG) — LLM identifies causal/sequential relations between log entries, forms subgraphs, and summarizes each into a situation description; (2) Topic Outline (MT) — LLM segments each dialogue into topics, extracts {requirement, solutions+feedback, preference} schema, and rewrites requirements enriched with retrieved log situations. - Abstract level: (3) Background (MB) — recursive cross-session summarization of situation entries across fixed aspects (work, family, etc.); (4) Principle (MP) — KMeans clustering of all historical requirements, with LLM extracting abstract requirement types and preference principles per cluster; updated incrementally for new sessions.

RAG: At inference, top-k entries (k=3, cosine similarity via paraphrase-multilingual-mpnet-base-v2) are retrieved from each memory tier; cross-tier mappings (FGB: MG→MB, FTP: MT→MP) propagate concrete retrievals to their abstract counterparts.

Key Contributions

  • PAL-Bench: first Chinese benchmark for long-term service-oriented user-agent interaction, with three evaluation tasks (Requirement Restatement, Solution Proposal, Multi-turn Dialogue Interaction)
  • PAL-Set: 100 synthetic users, avg. 29 sessions, 996 behavioral logs, 401 dialogue turns per user, spanning ~9.4 months; includes both logs and dialogues
  • H2 Memory: hierarchical heterogeneous memory with separate encodings for logs vs. dialogues and concrete vs. abstract levels, with incremental update mechanisms
  • Validation on external English dataset (LongMemEval) demonstrating cross-lingual generalizability

Results

Requirement Restatement (BLEU-1 / GPT-4 Score): - Vanilla (no history): 13.59 / 17.50 - Session-level RAG: 23.81 / 29.33 - MemoryBank (best prior): 23.89 / 28.57 - H2 Memory: 26.67 / 32.54 (best overall) - Ablation: removing MT causes the largest drop (24.04 / 28.00)

Solution Proposal (BLEU-1 / Selection Score): - Session-level RAG: 19.66 / 33.78 - MemoryBank (best prior): 20.49 / 29.85 - H2 Memory: 22.24 / 38.32 (best overall) - Ablation: removing MT causes the largest drop (18.23 / 28.09); removing MB hurts more than removing MG (preference is more abstract than requirement)

Multi-turn Dialogue Interaction (Win/Tie/Lose vs. baselines): - vs. Vanilla (w/o log): 478/29/319 (Requirement), 480/18/328 (Preference) - vs. MemoryBank: 449/33/344 / 452/25/349 - H2 Memory consistently wins across requirement and preference dimensions against all baselines

LongMemEval (single-session-preference, Accuracy): - Vanilla: 10.00%; MemoryBank: 23.33%; ConditionMem: 40.00% - H2 Memory (MT+MP): 50.00%

Limitations

  • Dataset is entirely synthetic (LLM-generated), which may not fully capture real-world behavioral diversity and conversational messiness
  • Only 100 users, limiting statistical power and persona diversity
  • External validation (LongMemEval) uses only the dialogue components (MT+MP) since that dataset lacks logs; log-based memory components are not evaluated cross-dataset
  • Multi-turn evaluation relies on LLM-as-judge and User-LLM simulation, introducing randomness that contributes to non-negligible "Lose" cases even against simple baselines
  • RAG relies on embedding similarity, which may misretrieve semantically similar but contextually irrelevant memories
  • Compute cost of the multi-stage memory construction pipeline (graph building, clustering, recursive summarization) is not analyzed

Relevance to Agentic AI / LLM Agents

This work directly addresses the long-term personalization challenge in deployed LLM agents—specifically how agents should maintain and update structured user models across dozens of sessions involving heterogeneous input modalities (behavioral logs + dialogue). The H2 Memory architecture exemplifies a key design pattern for agentic systems: separating episodic (concrete, session-specific) from semantic (abstract, cross-session) memory, with explicit cross-tier retrieval linkages. PAL-Bench also provides a missing evaluation resource for assessing agents on preference modeling and implicit need inference—capabilities that go beyond factual recall and are essential for real-world assistants. The incremental update mechanisms for MP and MB offer a practical template for continual personalization without full reprocessing.