PersonaVLM: Long-Term Personalized Multimodal LLMs

• URL: http://arxiv.org/abs/2604.13074v1
• Date: Fri, 20 Mar 2026 17:59:57 GMT
• Title: PersonaVLM: Long-Term Personalized Multimodal LLMs
• Abstract summary: PersonaVLM is an innovative personalized multimodal agent framework for long-term personalization.<n>It transforms a general-purpose MLLM into a personalized assistant by integrating three key capabilities.<n>It infers the user's evolving personality throughout long-term interactions to ensure outputs remain aligned with their unique characteristics.
• Score: 32.13357178547938
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multimodal Large Language Models (MLLMs) serve as daily assistants for millions. However, their ability to generate responses aligned with individual preferences remains limited. Prior approaches enable only static, single-turn personalization through input augmentation or output alignment, and thus fail to capture users' evolving preferences and personality over time (see Fig.1). In this paper, we introduce PersonaVLM, an innovative personalized multimodal agent framework designed for long-term personalization. It transforms a general-purpose MLLM into a personalized assistant by integrating three key capabilities: (a) Remembering: It proactively extracts and summarizes chronological multimodal memories from interactions, consolidating them into a personalized database. (b) Reasoning: It conducts multi-turn reasoning by retrieving and integrating relevant memories from the database. (c) Response Alignment: It infers the user's evolving personality throughout long-term interactions to ensure outputs remain aligned with their unique characteristics. For evaluation, we establish Persona-MME, a comprehensive benchmark comprising over 2,000 curated interaction cases, designed to assess long-term MLLM personalization across seven key aspects and 14 fine-grained tasks. Extensive experiments validate our method's effectiveness, improving the baseline by 22.4% (Persona-MME) and 9.8% (PERSONAMEM) under a 128k context, while outperforming GPT-4o by 5.2% and 2.0%, respectively. Project page: https://PersonaVLM.github.io.

Related papers

• Synthetic Interaction Data for Scalable Personalization in Large Language Models [67.31884245564086]
  We introduce a high-fidelity synthetic data generation framework called PersonaGym.<n>Unlike prior work that treats personalization as static persona-preference pairs, PersonaGym models a dynamic preference process.<n>We release PersonaAtlas, a large-scale, high-quality, and diverse synthetic dataset of high-fidelity multi-turn personalized interaction trajectories.
  arXiv  Detail & Related papers  (2026-02-12T20:41:22Z)
• Rethinking Personalization in Large Language Models at the Token Level [89.64259958702165]
  Personalization is typically framed as an additional layer on top of a base NLP task.<n>From a token-level perspective, different tokens in a response contribute to personalization to varying degrees.<n>We propose PerContrast, a self-contrast method that estimates each output token's dependence on user-specific information.
  arXiv  Detail & Related papers  (2026-02-04T16:05:02Z)
• MMPB: It's Time for Multi-Modal Personalization [7.876682671110321]
  We introduce MMPB, the first extensive benchmark for evaluating Vision-Language Models (VLMs) on personalization.<n>MMPB comprises 10k image-query pairs and includes 111 personalizable concepts across four categories: humans, animals, objects, and characters.<n>We evaluate personalization performance via a three-stage protocol: concept injection, multi-turn dialogue, and personalized querying.
  arXiv  Detail & Related papers  (2025-09-26T18:24:48Z)
• A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations [112.81207927088117]
  PersonaConvBench is a benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs)<n>We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements.
  arXiv  Detail & Related papers  (2025-05-20T09:13:22Z)
• Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale [53.059480071818136]
  Large Language Models (LLMs) have emerged as personalized assistants for users across a wide range of tasks.<n> PERSONAMEM features curated user profiles with over 180 simulated user-LLM interaction histories.<n>We evaluate LLM chatbots' ability to identify the most suitable response according to the current state of the user's profile.
  arXiv  Detail & Related papers  (2025-04-19T08:16:10Z)
• RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models [53.304699445700926]
  We introduce the Retrieval Augmented Personalization framework for MLLMs' personalization.<n>Starting from a general MLLM, we turn it into a personalized assistant in three steps.<n>By pretraining on large-scale dataset, RAP-MLLMs can generalize to infinite visual concepts without additional finetuning.
  arXiv  Detail & Related papers  (2024-10-17T09:10:26Z)
• Aligning LLMs with Individual Preferences via Interaction [51.72200436159636]
  We train large language models (LLMs) that can ''interact to align''<n>We develop a multi-turn preference dataset containing 3K+ multi-turn conversations in tree structures.<n>For evaluation, we establish the ALOE benchmark, consisting of 100 carefully selected examples and well-designed metrics to measure the customized alignment performance during conversations.
  arXiv  Detail & Related papers  (2024-10-04T17:48:29Z)
• PEFT-U: Parameter-Efficient Fine-Tuning for User Personalization [9.594958534074074]
  We introduce the PEFT-U Benchmark: a new dataset for building and evaluating NLP models for user personalization. We explore the challenge of efficiently personalizing LLMs to accommodate user-specific preferences in the context of diverse user-centered tasks.
  arXiv  Detail & Related papers  (2024-07-25T14:36:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.