ViPer: Visual Personalization of Generative Models via Individual Preference Learning

• URL: http://arxiv.org/abs/2407.17365v1
• Date: Wed, 24 Jul 2024 15:42:34 GMT
• Title: ViPer: Visual Personalization of Generative Models via Individual Preference Learning
• Authors: Sogand Salehi, Mahdi Shafiei, Teresa Yeo, Roman Bachmann, Amir Zamir,
• Abstract summary: We propose to personalize the image generation process by capturing the generic preferences of the user in a one-time process. Based on these comments, we infer a user's structured liked and disliked visual attributes. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference.
• Score: 11.909247529297678
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Different users find different images generated for the same prompt desirable. This gives rise to personalized image generation which involves creating images aligned with an individual's visual preference. Current generative models are, however, unpersonalized, as they are tuned to produce outputs that appeal to a broad audience. Using them to generate images aligned with individual users relies on iterative manual prompt engineering by the user which is inefficient and undesirable. We propose to personalize the image generation process by first capturing the generic preferences of the user in a one-time process by inviting them to comment on a small selection of images, explaining why they like or dislike each. Based on these comments, we infer a user's structured liked and disliked visual attributes, i.e., their visual preference, using a large language model. These attributes are used to guide a text-to-image model toward producing images that are tuned towards the individual user's visual preference. Through a series of user studies and large language model guided evaluations, we demonstrate that the proposed method results in generations that are well aligned with individual users' visual preferences.

Related papers

• ComPO: Community Preferences for Language Model Personalization [122.54846260663922]
  ComPO is a method to personalize preference optimization in language models. We collect and release ComPRed, a question answering dataset with community-level preferences from Reddit.
  arXiv  Detail & Related papers  (2024-10-21T14:02:40Z)
• JeDi: Joint-Image Diffusion Models for Finetuning-Free Personalized Text-to-Image Generation [49.997839600988875]
  Existing personalization methods rely on finetuning a text-to-image foundation model on a user's custom dataset. We propose Joint-Image Diffusion (jedi), an effective technique for learning a finetuning-free personalization model. Our model achieves state-of-the-art generation quality, both quantitatively and qualitatively, significantly outperforming both the prior finetuning-based and finetuning-free personalization baselines.
  arXiv  Detail & Related papers  (2024-07-08T17:59:02Z)
• User-Friendly Customized Generation with Multi-Modal Prompts [21.873076466803145]
  We propose a novel integration of text and images tailored to each customization concept. Our proposed paradigm for customized text-to-image generation surpasses existing finetune-based methods in user-friendliness.
  arXiv  Detail & Related papers  (2024-05-26T09:34:16Z)
• U-VAP: User-specified Visual Appearance Personalization via Decoupled Self Augmentation [18.841473623776153]
  State-of-the-art personalization models tend to overfit the whole subject and cannot disentangle visual characteristics in pixel space. A novel decoupled self-augmentation strategy is proposed to generate target-related and non-target samples to learn user-specified visual attributes. Experiments on various kinds of visual attributes with SOTA personalization methods show the ability of the proposed method to mimic target visual appearance in novel contexts.
  arXiv  Detail & Related papers  (2024-03-29T15:20:34Z)
• Learning User Embeddings from Human Gaze for Personalised Saliency Prediction [12.361829928359136]
  We present a novel method to extract user embeddings from pairs of natural images and corresponding saliency maps. At the core of our method is a Siamese convolutional neural encoder that learns the user embeddings by contrasting the image and personal saliency map pairs of different users.
  arXiv  Detail & Related papers  (2024-03-20T14:58:40Z)
• Personalized Language Modeling from Personalized Human Feedback [49.344833339240566]
  Reinforcement Learning from Human Feedback (RLHF) is commonly used to fine-tune large language models to better align with human preferences. In this work, we aim to address this problem by developing methods for building personalized language models.
  arXiv  Detail & Related papers  (2024-02-06T04:18:58Z)
• Pick-and-Draw: Training-free Semantic Guidance for Text-to-Image Personalization [56.12990759116612]
  Pick-and-Draw is a training-free semantic guidance approach to boost identity consistency and generative diversity for personalization methods. The proposed approach can be applied to any personalized diffusion models and requires as few as a single reference image.
  arXiv  Detail & Related papers  (2024-01-30T05:56:12Z)
• User-Aware Prefix-Tuning is a Good Learner for Personalized Image Captioning [35.211749514733846]
  Traditional image captioning methods often overlook the preferences and characteristics of users. Most existing methods emphasize the user context fusion process by memory networks or transformers. We propose a novel personalized image captioning framework that leverages user context to consider personality factors.
  arXiv  Detail & Related papers  (2023-12-08T02:08:00Z)
• Identity Encoder for Personalized Diffusion [57.1198884486401]
  We propose an encoder-based approach for personalization. We learn an identity encoder which can extract an identity representation from a set of reference images of a subject. We show that our approach consistently outperforms existing fine-tuning based approach in both image generation and reconstruction.
  arXiv  Detail & Related papers  (2023-04-14T23:32:24Z)
• Generating Person Images with Appearance-aware Pose Stylizer [66.44220388377596]
  We present a novel end-to-end framework to generate realistic person images based on given person poses and appearances. The core of our framework is a novel generator called Appearance-aware Pose Stylizer (APS) which generates human images by coupling the target pose with the conditioned person appearance progressively.
  arXiv  Detail & Related papers  (2020-07-17T15:58:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.