Contextualized Visual Personalization in Vision-Language Models

• URL: http://arxiv.org/abs/2602.03454v1
• Date: Tue, 03 Feb 2026 12:21:26 GMT
• Title: Contextualized Visual Personalization in Vision-Language Models
• Authors: Yeongtak Oh, Sangwon Yu, Junsung Park, Han Cheol Moon, Jisoo Mok, Sungroh Yoon,
• Abstract summary: We propose a unified framework that treats personalized image captioning as a core task for contextualized visual personalization.<n>In experiments, CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks.<n>These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.
• Score: 51.3151397451851
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite recent progress in vision-language models (VLMs), existing approaches often fail to generate personalized responses based on the user's specific experiences, as they lack the ability to associate visual inputs with a user's accumulated visual-textual context. We newly formalize this challenge as contextualized visual personalization, which requires the visual recognition and textual retrieval of personalized visual experiences by VLMs when interpreting new images. To address this issue, we propose CoViP, a unified framework that treats personalized image captioning as a core task for contextualized visual personalization and improves this capability through reinforcement-learning-based post-training and caption-augmented generation. We further introduce diagnostic evaluations that explicitly rule out textual shortcut solutions and verify whether VLMs truly leverage visual context. Extensive experiments demonstrate that existing open-source and proprietary VLMs exhibit substantial limitations, while CoViP not only improves personalized image captioning but also yields holistic gains across downstream personalization tasks. These results highlight CoViP as a crucial stage for enabling robust and generalizable contextualized visual personalization.

Related papers

• Multimodal Prompt Alignment for Facial Expression Recognition [24.470095812039286]
  MPA-FER provides fine-grained semantic guidance to the learning process of prompted visual features.<n>Our framework outperforms state-of-the-art methods on three FER benchmark datasets.
  arXiv  Detail & Related papers  (2025-06-26T05:28:57Z)
• Revisit What You See: Disclose Language Prior in Vision Tokens for LVLM Decoding [6.612630497074871]
  Large Vision-Language Models (LVLMs) achieve strong performance across multimodal tasks by integrating visual perception with language understanding.<n>We propose ReVisiT, a training-free decoding method that references vision tokens to guide text generation.
  arXiv  Detail & Related papers  (2025-06-11T08:46:55Z)
• Exploring Multimodal Prompt for Visualization Authoring with Large Language Models [12.43647167483504]
  We study how large language models (LLMs) interpret ambiguous or incomplete text prompts in the context of visualization authoring.<n>We introduce visual prompts as a complementary input modality to text prompts, which help clarify user intent.<n>We design VisPilot, which enables users to easily create visualizations using multimodal prompts, including text, sketches, and direct manipulations.
  arXiv  Detail & Related papers  (2025-04-18T14:00:55Z)
• Teaching VLMs to Localize Specific Objects from In-context Examples [56.797110842152]
  We find that present-day Vision-Language Models (VLMs) lack a fundamental cognitive ability: learning to localize specific objects in a scene by taking into account the context.<n>This work is the first to explore and benchmark personalized few-shot localization for VLMs.
  arXiv  Detail & Related papers  (2024-11-20T13:34:22Z)
• Enhancing Visual Document Understanding with Contrastive Learning in Large Visual-Language Models [56.76307866160105]
  We propose a contrastive learning framework, termed Document Object COntrastive learning (DoCo) DoCo leverages an auxiliary multimodal encoder to obtain the features of document objects and align them to the visual features generated by the vision encoder of Large Visual-Language Models (LVLMs) We demonstrate that the proposed DoCo serves as a plug-and-play pre-training method, which can be employed in the pre-training of various LVLMs without inducing any increase in computational complexity during the inference process.
  arXiv  Detail & Related papers  (2024-02-29T10:17:27Z)
• Improving In-Context Learning in Diffusion Models with Visual Context-Modulated Prompts [83.03471704115786]
  We introduce improved Prompt Diffusion (iPromptDiff) in this study. iPromptDiff integrates an end-to-end trained vision encoder that converts visual context into an embedding vector. We show that a diffusion-based vision foundation model, when equipped with this visual context-modulated text guidance and a standard ControlNet structure, exhibits versatility and robustness across a variety of training tasks.
  arXiv  Detail & Related papers  (2023-12-03T14:15:52Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)
• Visually-augmented pretrained language models for NLP tasks without images [77.74849855049523]
  Existing solutions often rely on explicit images for visual knowledge augmentation. We propose a novel textbfVisually-textbfAugmented fine-tuning approach. Our approach can consistently improve the performance of BERT, RoBERTa, BART, and T5 at different scales.
  arXiv  Detail & Related papers  (2022-12-15T16:13:25Z)
• SgVA-CLIP: Semantic-guided Visual Adapting of Vision-Language Models for Few-shot Image Classification [84.05253637260743]
  We propose a new framework, named Semantic-guided Visual Adapting (SgVA), to extend vision-language pre-trained models. SgVA produces discriminative task-specific visual features by comprehensively using a vision-specific contrastive loss, a cross-modal contrastive loss, and an implicit knowledge distillation. State-of-the-art results on 13 datasets demonstrate that the adapted visual features can well complement the cross-modal features to improve few-shot image classification.
  arXiv  Detail & Related papers  (2022-11-28T14:58:15Z)
• Learning Visual Representations with Caption Annotations [19.24013129952071]
  We propose a proxy task to learn visual representations over image-caption pairs. ICMLM consists in predicting masked words in captions by relying on visual cues. Our experiments confirm that image captions can be leveraged to inject global and localized semantic information into visual representations.
  arXiv  Detail & Related papers  (2020-08-04T08:04:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.