ViPer: Visual Personalization of Generative Models via Individual Preference Learning

• URL: http://arxiv.org/abs/2407.17365v1
• Date: Wed, 24 Jul 2024 15:42:34 GMT
• Title: ViPer: Visual Personalization of Generative Models via Individual Preference Learning
• Authors: Sogand Salehi, Mahdi Shafiei, Teresa Yeo, Roman Bachmann, Amir Zamir,
• Abstract summary: We propose to personalize the image generation process by capturing the generic preferences of the user in a one-time process. Based on these comments, we infer a user's structured liked and disliked visual attributes. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference.
• Score: 11.909247529297678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Different users find different images generated for the same prompt desirable. This gives rise to personalized image generation which involves creating images aligned with an individual's visual preference. Current generative models are, however, unpersonalized, as they are tuned to produce outputs that appeal to a broad audience. Using them to generate images aligned with individual users relies on iterative manual prompt engineering by the user which is inefficient and undesirable. We propose to personalize the image generation process by first capturing the generic preferences of the user in a one-time process by inviting them to comment on a small selection of images, explaining why they like or dislike each. Based on these comments, we infer a user's structured liked and disliked visual attributes, i.e., their visual preference, using a large language model. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference. Through a series of user studies and large language model guided evaluations, we demonstrate that the proposed method results in generations that are well aligned with individual users' visual preferences.

Related papers

• DRC: Enhancing Personalized Image Generation via Disentangled Representation Composition [69.10628479553709]
  We introduce DRC, a novel personalized image generation framework that enhances Large Multimodal Models (LMMs) DRC explicitly extracts user style preferences and semantic intentions from history images and the reference image, respectively. It involves two critical learning stages: 1) Disentanglement learning, which employs a dual-tower disentangler to explicitly separate style and semantic features, optimized via a reconstruction-driven paradigm with difficulty-aware importance sampling; and 2) Personalized modeling, which applies semantic-preserving augmentations to effectively adapt the disentangled representations for robust personalized generation.
  arXiv  Detail & Related papers  (2025-04-24T08:10:10Z)
• Enhancing Intent Understanding for Ambiguous prompt: A Human-Machine Co-Adaption Strategy [28.647935556492957]
  We propose a human-machine co-adaption strategy using mutual information between the user's prompts and the pictures under modification. We find that an improved model can reduce the necessity for multiple rounds of adjustments.
  arXiv  Detail & Related papers  (2025-01-25T10:32:00Z)
• Personalized Preference Fine-tuning of Diffusion Models [75.22218338096316]
  We introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way. Our approach achieves an average win rate of 76% over Stable Cascade, generating images that more accurately reflect specific user preferences.
  arXiv  Detail & Related papers  (2025-01-11T22:38:41Z)
• ComPO: Community Preferences for Language Model Personalization [122.54846260663922]
  ComPO is a method to personalize preference optimization in language models. We collect and release ComPRed, a question answering dataset with community-level preferences from Reddit.
  arXiv  Detail & Related papers  (2024-10-21T14:02:40Z)
• Personalized Image Generation with Large Multimodal Models [47.289887243367055]
  We propose a Personalized Image Generation Framework named Pigeon to capture users' visual preferences and needs from noisy user history and multimodal instructions. We apply Pigeon to personalized sticker and movie poster generation, where extensive quantitative results and human evaluation highlight its superiority over various generative baselines.
  arXiv  Detail & Related papers  (2024-10-18T04:20:46Z)
• JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [49.997839600988875]
  Existing personalization methods rely on finetuning a text-to-image foundation model on a user's custom dataset. We propose Joint-Image Diffusion (jedi), an effective technique for learning a finetuning-free personalization model. Our model achieves state-of-the-art generation quality, both quantitatively and qualitatively, significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.
  arXiv  Detail & Related papers  (2024-07-08T17:59:02Z)
• U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation [18.841473623776153]
  State-of-the-art personalization models tend to overfit the whole subject and cannot disentangle visual characteristics in pixel space. A novel decoupled self-augmentation strategy is proposed to generate target-related and non-target samples to learn user-specified visual attributes. Experiments on various kinds of visual attributes with SOTA personalization methods show the ability of the proposed method to mimic target visual appearance in novel contexts.
  arXiv  Detail & Related papers  (2024-03-29T15:20:34Z)
• Learning User Embeddings from Human Gaze for Personalised Saliency Prediction [12.361829928359136]
  We present a novel method to extract user embeddings from pairs of natural images and corresponding saliency maps. At the core of our method is a Siamese convolutional neural encoder that learns the user embeddings by contrasting the image and personal saliency map pairs of different users.
  arXiv  Detail & Related papers  (2024-03-20T14:58:40Z)
• Personalized Language Modeling from Personalized Human Feedback [49.344833339240566]
  Reinforcement Learning from Human Feedback (RLHF) is commonly used to fine-tune large language models to better align with human preferences. In this work, we aim to address this problem by developing methods for building personalized language models.
  arXiv  Detail & Related papers  (2024-02-06T04:18:58Z)
• Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization [56.12990759116612]
  Pick-and-Draw is a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image.
  arXiv  Detail & Related papers  (2024-01-30T05:56:12Z)
• User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning [35.211749514733846]
  Traditional image captioning methods often overlook the preferences and characteristics of users. Most existing methods emphasize the user context fusion process by memory networks or transformers. We propose a novel personalized image captioning framework that leverages user context to consider personality factors.
  arXiv  Detail & Related papers  (2023-12-08T02:08:00Z)
• Identity Encoder for Personalized Diffusion [57.1198884486401]
  We propose an encoder-based approach for personalization. We learn an identity encoder which can extract an identity representation from a set of reference images of a subject. We show that our approach consistently outperforms existing fine-tuning based approach in both image generation and reconstruction.
  arXiv  Detail & Related papers  (2023-04-14T23:32:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.