Ego: Embedding-Guided Personalization of Vision-Language Models

• URL: http://arxiv.org/abs/2603.09771v2
• Date: Wed, 11 Mar 2026 15:26:01 GMT
• Title: Ego: Embedding-Guided Personalization of Vision-Language Models
• Abstract summary: We propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts.<n>Specifically, we extract visual tokens that predominantly represent the target concept.<n>These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images.
• Score: 8.63838537704658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI assistants that support humans in daily life are becoming increasingly feasible, driven by the rapid advancements in multimodal language models. A key challenge lies in overcoming the generic nature of these models to deliver personalized experiences. Existing approaches to personalizing large vision language models often rely on additional training stages, which limit generality and scalability, or on engineered pipelines with external pre-trained modules, which hinder deployment efficiency. In this work, we propose an efficient personalization method that leverages the model's inherent ability to capture personalized concepts. Specifically, we extract visual tokens that predominantly represent the target concept by utilizing the model's internal attention mechanisms. These tokens serve as a memory of that specific concept, enabling the model to recall and describe it when it appears in test images. We conduct a comprehensive and unified evaluation of our approach and SOTA methods across various personalization settings including single-concept, multi-concept, and video personalization, demonstrating strong performance gains with minimal personalization overhead.

Related papers

• CPC-VAR:Continual Personalized and Compositional Generation in Visual Autoregressive Models [69.9266117908314]
  Visual autoregressive ( VAR) models have emerged as efficient paradigm for text-to-image generation.<n>Existing VAR-based personalization methods fail to accommodate evolving user demands.<n>We present the first systematic study of continual personalized generation in VAR models.
  arXiv  Detail & Related papers  (2026-05-19T12:18:15Z)
• Zero-Shot Personalization of Objects via Textual Inversion [36.82648022347006]
  We propose a novel framework that employs a learned network to predict object-specific textual embeddings.<n>This design enables rapid, zero-shot personalization of a wide range of objects in a single forward pass.<n>To the best of our knowledge, this work represents the first attempt to achieve such general-purpose, training-free personalization within diffusion models.
  arXiv  Detail & Related papers  (2026-03-24T09:54:30Z)
• Unified Personalized Understanding, Generating and Editing [54.5563878110386]
  We present textbf OmniPersona, an end-to-end personalization framework for unified LMMs.<n>It integrates personalized understanding, generation, and image editing within a single architecture.<n>Experiments demonstrate that OmniPersona delivers competitive and robust performance across diverse personalization tasks.
  arXiv  Detail & Related papers  (2026-01-11T15:46:34Z)
• MC-LLaVA: Multi-Concept Personalized Vision-Language Model [51.645660375766575]
  This paper proposes the first multi-concept personalization paradigm, MC-LLaVA.<n>MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step.<n> Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses.
  arXiv  Detail & Related papers  (2025-03-24T16:32:17Z)
• Personalization Toolkit: Training Free Personalization of Large Vision Language Models [11.026377387506216]
  Personalization of Large Vision-Language Models (LVLMs) involves customizing models to recognize specific users and object instances, and to generate contextually tailored responses.<n>Existing approaches typically rely on time-consuming test-time training for each user or object, making them impractical for real-world deployment.<n>We present a novel training-free approach to LVLM personalization and introduce a comprehensive real-world benchmark designed to rigorously evaluate various aspects of the personalization task.
  arXiv  Detail & Related papers  (2025-02-04T16:19:20Z)
• MC-LLaVA: Multi-Concept Personalized Vision-Language Model [51.645660375766575]
  This paper proposes the first multi-concept personalization paradigm, MC-LLaVA.<n>MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step.<n> Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses.
  arXiv  Detail & Related papers  (2024-11-18T16:33:52Z)
• Personalized Visual Instruction Tuning [30.677058613937067]
  multimodal large language models (MLLMs) can engage in general conversations but fail to conduct personalized dialogues targeting at specific individuals. This deficiency hinders the application of MLLMs in personalized settings, such as tailored visual assistants on mobile devices. We introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image.
  arXiv  Detail & Related papers  (2024-10-09T17:46:53Z)
• Enhancing Large Vision Language Models with Self-Training on Image Comprehension [131.14381425260706]
  We introduce Self-Training on Image (STIC), which emphasizes a self-training approach specifically for image comprehension. First, the model self-constructs a preference for image descriptions using unlabeled images. To further self-improve reasoning on the extracted visual information, we let the model reuse a small portion of existing instruction-tuning data.
  arXiv  Detail & Related papers  (2024-05-30T05:53:49Z)
• Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond [87.1712108247199]
  Our goal is to establish a Unified paradigm for Multi-modal Personalization systems (UniMP) We develop a generic and personalization generative framework, that can handle a wide range of personalized needs. Our methodology enhances the capabilities of foundational language models for personalized tasks.
  arXiv  Detail & Related papers  (2024-03-15T20:21:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.