Retrieving Conditions from Reference Images for Diffusion Models

• URL: http://arxiv.org/abs/2312.02521v2
• Date: Fri, 15 Mar 2024 04:37:32 GMT
• Title: Retrieving Conditions from Reference Images for Diffusion Models
• Authors: Haoran Tang, Xin Zhou, Jieren Deng, Zhihong Pan, Hao Tian, Pratik Chaudhari,
• Abstract summary: We consider Subject-Driven Generation as a unified retrieval problem with diffusion models. We introduce a novel diffusion model architecture, named RetriNet, designed to address and solve these problems. We also propose a research and friendly dataset, RetriBooru, to study a more difficult problem, concept composition.
• Score: 30.14303302029618
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Newly developed diffusion-based techniques have showcased phenomenal abilities in producing a wide range of high-quality images, sparking considerable interest in various applications. A prevalent scenario is to generate new images based on a subject from reference images. This subject could be face identity for styled avatars, body and clothing for virtual try-on and so on. Satisfying this requirement is evolving into a field called Subject-Driven Generation. In this paper, we consider Subject-Driven Generation as a unified retrieval problem with diffusion models. We introduce a novel diffusion model architecture, named RetriNet, designed to address and solve these problems by retrieving subject attributes from reference images precisely, and filter out irrelevant information. RetriNet demonstrates impressive performance when compared to existing state-of-the-art approaches in face generation. We further propose a research and iteration friendly dataset, RetriBooru, to study a more difficult problem, concept composition. Finally, to better evaluate alignment between similarity and diversity or measure diversity that have been previously unaccounted for, we introduce a novel class of metrics named Similarity Weighted Diversity (SWD).

Related papers

• Group-based Distinctive Image Captioning with Memory Difference Encoding and Attention [62.246950834745796]
  Group-based Differential Distinctive Captioning Method. Group-based Differential Memory Attention (GDMA) module. New evaluation metric, the Distinctive Word Rate (DisWordRate)
  arXiv  Detail & Related papers  (2025-04-03T11:19:51Z)
• Fusion is all you need: Face Fusion for Customized Identity-Preserving Image Synthesis [7.099258248662009]
  Text-to-image (T2I) models have significantly advanced the development of artificial intelligence. However, existing T2I-based methods often struggle to accurately reproduce the appearance of individuals from a reference image. We leverage the pre-trained UNet from Stable Diffusion to incorporate the target face image directly into the generation process.
  arXiv  Detail & Related papers  (2024-09-27T19:31:04Z)
• Revolutionizing Text-to-Image Retrieval as Autoregressive Token-to-Voken Generation [90.71613903956451]
  Text-to-image retrieval is a fundamental task in multimedia processing. We propose an autoregressive voken generation method, named AVG. We show that AVG achieves superior results in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2024-07-24T13:39:51Z)
• Synthesizing Efficient Data with Diffusion Models for Person Re-Identification Pre-Training [51.87027943520492]
  We present a novel paradigm Diffusion-ReID to efficiently augment and generate diverse images based on known identities. Benefiting from our proposed paradigm, we first create a new large-scale person Re-ID dataset Diff-Person, which consists of over 777K images from 5,183 identities.
  arXiv  Detail & Related papers  (2024-06-10T06:26:03Z)
• When StyleGAN Meets Stable Diffusion: a $\mathscr{W}_+$ Adapter for Personalized Image Generation [60.305112612629465]
  Text-to-image diffusion models have excelled in producing diverse, high-quality, and photo-realistic images. We present a novel use of the extended StyleGAN embedding space $mathcalW_+$ to achieve enhanced identity preservation and disentanglement for diffusion models. Our method adeptly generates personalized text-to-image outputs that are not only compatible with prompt descriptions but also amenable to common StyleGAN editing directions.
  arXiv  Detail & Related papers  (2023-11-29T09:05:14Z)
• DiffusePast: Diffusion-based Generative Replay for Class Incremental Semantic Segmentation [73.54038780856554]
  Class Incremental Semantic (CISS) extends the traditional segmentation task by incrementally learning newly added classes. Previous work has introduced generative replay, which involves replaying old class samples generated from a pre-trained GAN. We propose DiffusePast, a novel framework featuring a diffusion-based generative replay module that generates semantically accurate images with more reliable masks guided by different instructions.
  arXiv  Detail & Related papers  (2023-08-02T13:13:18Z)
• Break-A-Scene: Extracting Multiple Concepts from a Single Image [80.47666266017207]
  We introduce the task of textual scene decomposition. We propose augmenting the input image with masks that indicate the presence of target concepts. We then present a novel two-phase customization process.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Pseudo-Pair based Self-Similarity Learning for Unsupervised Person Re-identification [47.44945334929426]
  We present a pseudo-pair based self-similarity learning approach for unsupervised person re-ID without human annotations. We propose to assign pseudo labels to images through the pairwise-guided similarity separation. It learns local discriminative features from individual images via intra-similarity, and discovers the patch correspondence across images via inter-similarity.
  arXiv  Detail & Related papers  (2022-07-09T04:05:06Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Semantic Diversity Learning for Zero-Shot Multi-label Classification [14.480713752871523]
  This study introduces an end-to-end model training for multi-label zero-shot learning. We propose to use an embedding matrix having principal embedding vectors trained using a tailored loss function. In addition, during training, we suggest up-weighting in the loss function image samples presenting higher semantic diversity to encourage the diversity of the embedding matrix.
  arXiv  Detail & Related papers  (2021-05-12T19:39:07Z)
• Person image generation with semantic attention network for person re-identification [9.30413920076019]
  We propose a novel person pose-guided image generation method, which is called the semantic attention network. The network consists of several semantic attention blocks, where each block attends to preserve and update the pose code and the clothing textures. Compared with other methods, our network can characterize better body shape and keep clothing attributes, simultaneously.
  arXiv  Detail & Related papers  (2020-08-18T12:18:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.