PersonaVLM: Long-Term Personalized Multimodal LLMs¶

🕒 Published (v1): 2026-03-20 17:59 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PersonaVLM is an agentic framework that wraps a base MLLM (Qwen2.5-VL-7B) with a structured memory architecture and a two-stage Response/Update loop to achieve long-term, dynamic personalization. It tracks user personality via a Big Five vector updated with exponential moving average (EMA) and retrieves from four typed memory stores during multi-step reasoning. On Persona-MME and PERSONAMEM, it outperforms GPT-4o and improves over a strong baseline by 22.4% and 9.8%, respectively.

Problem¶

Existing MLLM personalization is static: input-augmentation methods (Yo'LLaVA, RAP) cannot update stored user concepts as preferences evolve, and alignment-based methods (AlignXpert, PAS) encode fixed personality traits that become stale over long-horizon interactions. No prior system jointly handles dynamic memory management, multi-turn retrieval with temporal grounding, and continuously evolving personality alignment.

Method¶

PersonaVLM wraps a frozen/fine-tuned MLLM with two collaborative stages:

Personalized Memory Architecture stores (1) a Big Five personality profile \(\mathbf{p} \in \mathbb{R}^5\) and (2) four typed memory banks: core (fundamental user attributes, MemGPT-style), semantic (entity/relation facts), episodic (timestamped atomic dialogue events with keywords), and procedural (habits and recurring goals). All support CRUD operations; core and procedural retain only the latest version while semantic and episodic are stored chronologically.

Response Stage implements multi-step agentic retrieval (Eq. 1): given query \(Q_m\), context \(C_m\), and memory state \(M_{m-1}\), the model emits structured <think>, <retrieve>, and <answer> tokens. If information is insufficient, it outputs a retrieval condition specifying a time range and keywords; the system executes parallel top-\(k\) search across semantic, episodic, and procedural stores and feeds results back. This iterates until <answer> is emitted.

Update Stage (Eq. 2) runs asynchronously post-response. The Personality Evolving Mechanism (PEM) infers a per-turn personality vector \(\mathbf{p}'_m\) from the new query, then blends it with a cosine-decayed EMA: \(\mathbf{p}_m \leftarrow \lambda \cdot \mathbf{p}_{m-1} + (1-\lambda) \cdot \mathbf{p}'_m\), where \(\lambda\) starts low for rapid early adaptation and rises for stability. Memory is also updated: semantic after each turn, core/procedural after each session, episodic by topic segmentation.

Training uses two stages: (1) SFT on 78k synthetic samples covering memory CRUD and multi-step reasoning trajectories; (2) GRPO with a reward \(r_i = f_\text{acc} \cdot f_\text{cons} + 0.5 \cdot f_\text{format}\), judged by Qwen-3-30B-A3B zero-shot, with retrieval capped at 3 attempts per trajectory.

A data synthesis pipeline generates 30k+ multi-turn multimodal interactions across 500 personas (sampled from PersonaHub, enriched with Big Five traits), with >15% multimodal turns and probabilistic preference/personality drift to simulate longitudinal evolution.

Key Contributions¶

PersonaVLM agent framework unifying proactive remembering, agentic multi-step retrieval, and dynamic personality-aligned generation
Four-type multimodal memory database + Big Five personality profile with cosine-decayed EMA (PEM) for longitudinal personality tracking
Persona-MME benchmark: 2,000+ in-situ test cases across 7 dimensions, 14 fine-grained tasks, 200 personas, at 32k and 128k context lengths; benchmarks 10+ proprietary and open-source models
Synthetic training dataset (78k SFT samples, 30k+ interaction turns) with structured intermediate supervision for memory and reasoning steps

Results¶

+22.4% over 128k-context baseline on Persona-MME; +9.8% over baseline on PERSONAMEM (128k context)
+5.2% over GPT-4o on Persona-MME; +2.0% over GPT-4o on PERSONAMEM
On PERSONAMEM (32k), PersonaVLM outperforms GPT-4o, LLaMA-3.1-8B, InternVL3-8B, and Qwen2.5-VL-7B across the full task distribution (see Fig. 4)
Benchmarked tasks span: Shared Experiences, Background Knowledge, Scenario Recall, New Ideas, Proactive Recommendation, Aligned Preference Evolution, Revisit Preferences, and others across 7 core dimensions

Limitations¶

Evaluation is conducted on synthetic benchmarks generated by the same pipeline used for training data, risking benchmark contamination or distribution mismatch with real users
Big Five personality representation is a coarse proxy; real human personality is higher-dimensional and context-sensitive beyond OCEAN scores
PEM adapts to text/multimodal signals but cannot verify whether inferred personality shifts reflect genuine user change vs. situational variance
Maximum retrieval depth is capped at 3 iterations per response, which may be insufficient for very long histories
Backbone (Qwen2.5-VL-7B) is 7B scale; generalization to larger or different backbone families is not evaluated

Relevance to Harnesses / Meta-Harnesses¶

PersonaVLM is structurally a meta-harness: it orchestrates a base MLLM (Qwen2.5-VL-7B) through a deterministic two-stage pipeline (Response → Update) with typed tool calls, structured output tokens, and asynchronous state management — the canonical harness pattern applied to personalization. The agentic retrieval loop with <think>/<retrieve>/<answer> tags is a concrete instantiation of harness-controlled chain-of-thought, where the harness mediates between model outputs and external memory CRUD operations at each reasoning step. The data synthesis pipeline is itself a harness: it chains a persona generator, a long-conversation simulator (Seed1.6-thinking), and a structured supervision extractor to produce training data at scale. For researchers building harnesses, PersonaVLM demonstrates how to design reward shaping (GRPO with LLM-as-judge) and training curricula (SFT cold-start → RL refinement) specifically to instill reliable harness-compliant behavior in an otherwise general model.