PersonaVLM: Long-Term Personalized Multimodal LLMs¶

🕒 Published (v1): 2026-03-20 17:59 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; multi-turn long-term personalized preference alignment in multimodal LLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

PersonaVLM is an agent framework that transforms a general-purpose MLLM into a long-term personalized assistant by equipping it with structured multimodal memory, agentic retrieval, and a dynamic personality model. It introduces Persona-MME, a new benchmark for evaluating long-term multimodal personalization across 14 fine-grained tasks. The system improves over strong baselines by 22.4% on Persona-MME and 9.8% on PERSONAMEM, surpassing GPT-4o on both.

Problem¶

Existing MLLM personalization approaches are static and single-turn: input-augmentation methods (e.g., Yo'LLaVA, RAP) cannot update memories as user preferences shift, while alignment-based methods (e.g., PAS, ALIGNXPERT) presuppose fixed user traits and cannot adapt as personality evolves over long-term interaction. No prior benchmark holistically evaluates dynamic, multimodal, long-horizon personalization.

Method¶

PersonaVLM wraps Qwen2.5-VL-7B with a two-stage inference loop and a four-type memory database:

Memory architecture maintains (1) a personality profile \(\mathbf{p} \in \mathbb{R}^5\) encoding Big Five (OCEAN) trait scores, and (2) four memory types: core (fundamental user attributes), semantic (entity/relation facts), episodic (timestamped event summaries), and procedural (habits/goals). CRUD operations keep all types current.

Response Stage: Given query \(Q_m\), context \(C_m\), and memory state \(M_{m-1}\), the model iteratively outputs structured <think>, <retrieve>, or <answer> tokens. Retrieval is timeline-scoped (inferred time period + keywords) and searches semantic, episodic, and procedural memories in parallel, with top-\(k\) results fed back for the next reasoning step.

Update Stage: After each response, (1) a Personality Evolving Mechanism (PEM) infers turn-specific scores \(\mathbf{p}'_m\) and applies an EMA update \(\mathbf{p}_m \leftarrow \lambda \cdot \mathbf{p}_{m-1} + (1-\lambda) \cdot \mathbf{p}'_m\) with cosine-decayed \(\lambda\) for early plasticity and later stability; (2) memory types are selectively extracted and updated with tailored frequency (semantic per-turn; core/procedural per-session; episodic by topic segment).

Training is two-stage: SFT on 78k synthetic samples covering memory CRUD and multi-step QA, then GRPO reinforcement learning with a composite reward \(r_i = f_\text{acc} \cdot f_\text{cons} + 0.5 \cdot f_\text{format}\), judged by Qwen-30B-A3B.

A synthesis pipeline generates 30k+ interactions across 500 personas (>15% multimodal) with longitudinal preference/personality drift as training data.

Key Contributions¶

PersonaVLM framework: unified agent supporting proactive Remembering, multi-step Reasoning with timeline-scoped retrieval, and dynamic Response Alignment for long-term MLLM personalization.
Personalized Memory Architecture: four-type memory database (core, semantic, episodic, procedural) plus OCEAN-scored personality profile updated via PEM with EMA and cosine-decayed smoothing.
Persona-MME benchmark: 2,000+ in-situ cases, 200 personas, 7 core dimensions, 14 fine-grained tasks, two context configurations (32k and 128k); benchmarks 10+ models.
Synthetic training dataset: 30k+ multimodal long-horizon interactions with structured intermediate supervision (reasoning, retrieval, and memory operations).

Results¶

PersonaVLM improves over the Qwen2.5-VL-7B baseline by +22.4% on Persona-MME (128k context).
PersonaVLM improves over baseline by +9.8% on PERSONAMEM (128k context).
PersonaVLM outperforms GPT-4o by +5.2% on Persona-MME and +2.0% on PERSONAMEM.
On PERSONAMEM (32k), PersonaVLM outperforms LLaMA 3.1-8B, InternVL3-8B, Qwen2.5-VL-7B, and GPT-4o across the overall score and most sub-tasks (per Fig. 4).

Limitations¶

Backbone (Qwen2.5-VL-7B, 7B parameters) is modest; gains over GPT-4o are narrow (2–5%), suggesting larger backbones could yield further headroom.
Training data is fully synthetic, generated by a commercial model (Seed1.6-thinking); real-world distribution shift may affect deployment.
The memory system requires non-trivial inference-time compute: iterative multi-step retrieval with up to 3 rounds per response.
Persona-MME is itself derived from the same synthesis pipeline, which may favor PersonaVLM through distributional overlap.
Privacy analysis of on-device deployment is asserted but not experimentally validated.
Multimodal coverage in training data is limited to >15% of dialogues; heavy visual interaction scenarios may be underrepresented.

Relevance to Vision-Language Models¶

PersonaVLM directly extends the MLLM paradigm beyond single-turn, task-centric use toward persistent, user-adaptive assistants — a direction relevant to any VLM intended for prolonged human interaction. The framework's four-type multimodal memory architecture and agentic retrieval loop demonstrate how structured external memory can compensate for VLMs' fixed context constraints in real-world, longitudinal settings. Persona-MME fills a gap in VLM evaluation infrastructure, offering a benchmark that rewards dynamic understanding of visual and textual user history rather than static concept recognition. For researchers building personalized or embodied VLM agents, the PEM and GRPO-based reasoning training are directly reusable design patterns.