Personal Visual Context Learning in Large Multimodal Models

• URL: http://arxiv.org/abs/2605.10936v1
• Date: Mon, 11 May 2026 17:59:17 GMT
• Title: Personal Visual Context Learning in Large Multimodal Models
• Abstract summary: We present Personal Visual Context Learning (Personal ), the prompt-time capability of using user-specific visual context to resolve personalized queries.<n>Our analysis of LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied.<n>Motivated by these findings, we propose the Agentic Context Bank, a strong-time baseline that structures a user's visual context into a self-refining memory bank and employs query-adaptive evidence selection.
• Score: 53.33351452841322
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As wearable devices like smart glasses integrate Large Multimodal Models (LMMs) into the continuous first-person visual streams of individual users, the evolution of these models into true personal assistants hinges on visual personalization: the ability to reason over visual information unique to the wearer. We formalize this capability as Personal Visual Context Learning (Personal VCL), the prompt-time capability of using user-specific visual context to resolve personalized queries. To systematically evaluate this, we present Personal-VCL-Bench, a comprehensive benchmark capturing the personal visual world across persons, objects, and behaviors. Our analysis of frontier LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied. Motivated by these findings, we propose the Agentic Context Bank, a strong inference-time baseline that structures a user's visual context into a self-refining memory bank and employs query-adaptive evidence selection. Our baseline approach consistently improves over standard context prompting regimes across tasks and evaluated backbones, demonstrating a practical path towards future personalized LMMs.

Related papers

• Contextualized Visual Personalization in Vision-Language Models [51.3151397451851]
  We propose a unified framework that treats personalized image captioning as a core task for contextualized visual personalization.<n>In experiments, CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks.<n>These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.
  arXiv  Detail & Related papers  (2026-02-03T12:21:26Z)
• Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles [0.4885400580268118]
  Large language models (LLMs) increasingly serve as the central control unit of AI agents.<n>We present a framework that integrates persistent memory, dynamic coordination, self-validation, and evolving user profiles to enable personalized long-term interactions.
  arXiv  Detail & Related papers  (2025-10-09T08:22:16Z)
• Visual Representation Alignment for Multimodal Large Language Models [38.319869213758686]
  Multimodal large language models (MLLMs) trained with visual instruction tuning have achieved strong performance across diverse tasks.<n>But they remain limited in vision-centric tasks such as object counting or spatial reasoning.<n>We present VIsual Representation ALignment (VIRAL), a simple yet effective regularization strategy that aligns the internal visual representations of MLLMs with those of pre-trained vision foundation models.
  arXiv  Detail & Related papers  (2025-09-09T17:59:14Z)
• A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations [112.81207927088117]
  PersonaConvBench is a benchmark for evaluating personalized reasoning and generation in multi-turn conversations with large language models (LLMs)<n>We benchmark several commercial and open-source LLMs under a unified prompting setup and observe that incorporating personalized history yields substantial performance improvements.
  arXiv  Detail & Related papers  (2025-05-20T09:13:22Z)
• MC-LLaVA: Multi-Concept Personalized Vision-Language Model [51.645660375766575]
  This paper proposes the first multi-concept personalization paradigm, MC-LLaVA.<n>MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step.<n> Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses.
  arXiv  Detail & Related papers  (2025-03-24T16:32:17Z)
• Personalization Toolkit: Training Free Personalization of Large Vision Language Models [11.026377387506216]
  Personalization of Large Vision-Language Models (LVLMs) involves customizing models to recognize specific users and object instances, and to generate contextually tailored responses.<n>Existing approaches typically rely on time-consuming test-time training for each user or object, making them impractical for real-world deployment.<n>We present a novel training-free approach to LVLM personalization and introduce a comprehensive real-world benchmark designed to rigorously evaluate various aspects of the personalization task.
  arXiv  Detail & Related papers  (2025-02-04T16:19:20Z)
• MC-LLaVA: Multi-Concept Personalized Vision-Language Model [51.645660375766575]
  This paper proposes the first multi-concept personalization paradigm, MC-LLaVA.<n>MC-LLaVA employs a multi-concept instruction tuning strategy, effectively integrating multiple concepts in a single training step.<n> Comprehensive qualitative and quantitative experiments demonstrate that MC-LLaVA can achieve impressive multi-concept personalized responses.
  arXiv  Detail & Related papers  (2024-11-18T16:33:52Z)
• Personalized Visual Instruction Tuning [30.677058613937067]
  multimodal large language models (MLLMs) can engage in general conversations but fail to conduct personalized dialogues targeting at specific individuals. This deficiency hinders the application of MLLMs in personalized settings, such as tailored visual assistants on mobile devices. We introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image.
  arXiv  Detail & Related papers  (2024-10-09T17:46:53Z)
• Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement [102.22911097049953]
  Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks.<n>Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results.<n>We propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies.
  arXiv  Detail & Related papers  (2024-05-24T23:09:27Z)
• Multi-modal Auto-regressive Modeling via Visual Words [96.25078866446053]
  We propose the concept of visual tokens, which maps the visual features to probability distributions over Large Multi-modal Models' vocabulary. We further explore the distribution of visual features in the semantic space within LMM and the possibility of using text embeddings to represent visual information.
  arXiv  Detail & Related papers  (2024-03-12T14:58:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.