PluGeN: Multi-Label Conditional Generation From Pre-Trained Models

• URL: http://arxiv.org/abs/2109.09011v1
• Date: Sat, 18 Sep 2021 21:02:24 GMT
• Title: PluGeN: Multi-Label Conditional Generation From Pre-Trained Models
• Authors: Maciej Wo{\l}czyk, Magdalena Proszewska, {\L}ukasz Maziarka, Maciej Zi\k{e}ba, Patryk Wielopolski, Rafa{\l} Kurczab, Marek \'Smieja
• Abstract summary: PluGeN is a simple yet effective generative technique that can be used as a plugin to pre-trained generative models. We show that PluGeN preserves the quality of backbone models while adding the ability to control the values of labeled attributes.
• Score: 1.4777718769290524
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern generative models achieve excellent quality in a variety of tasks including image or text generation and chemical molecule modeling. However, existing methods often lack the essential ability to generate examples with requested properties, such as the age of the person in the photo or the weight of the generated molecule. Incorporating such additional conditioning factors would require rebuilding the entire architecture and optimizing the parameters from scratch. Moreover, it is difficult to disentangle selected attributes so that to perform edits of only one attribute while leaving the others unchanged. To overcome these limitations we propose PluGeN (Plugin Generative Network), a simple yet effective generative technique that can be used as a plugin to pre-trained generative models. The idea behind our approach is to transform the entangled latent representation using a flow-based module into a multi-dimensional space where the values of each attribute are modeled as an independent one-dimensional distribution. In consequence, PluGeN can generate new samples with desired attributes as well as manipulate labeled attributes of existing examples. Due to the disentangling of the latent representation, we are even able to generate samples with rare or unseen combinations of attributes in the dataset, such as a young person with gray hair, men with make-up, or women with beards. We combined PluGeN with GAN and VAE models and applied it to conditional generation and manipulation of images and chemical molecule modeling. Experiments demonstrate that PluGeN preserves the quality of backbone models while adding the ability to control the values of labeled attributes.

Related papers

• A Simple Approach to Unifying Diffusion-based Conditional Generation [63.389616350290595]
  We introduce a simple, unified framework to handle diverse conditional generation tasks. Our approach enables versatile capabilities via different inference-time sampling schemes. Our model supports additional capabilities like non-spatially aligned and coarse conditioning.
  arXiv  Detail & Related papers  (2024-10-15T09:41:43Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Conditional Generation from Unconditional Diffusion Models using Denoiser Representations [94.04631421741986]
  We propose adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of the denoiser network. We show that augmenting the Tiny ImageNet training set with synthetic images generated by our approach improves the classification accuracy of ResNet baselines by up to 8%.
  arXiv  Detail & Related papers  (2023-06-02T20:09:57Z)
• Generative Visual Prompt: Unifying Distributional Control of Pre-Trained Generative Models [77.47505141269035]
  Generative Visual Prompt (PromptGen) is a framework for distributional control over pre-trained generative models. PromptGen approximats an energy-based model (EBM) and samples images in a feed-forward manner. Code is available at https://github.com/ChenWu98/Generative-Visual-Prompt.
  arXiv  Detail & Related papers  (2022-09-14T22:55:18Z)
• Attribute Graphs Underlying Molecular Generative Models: Path to Learning with Limited Data [42.517927809224275]
  We provide an algorithm that relies on perturbation experiments on latent codes of a pre-trained generative autoencoder to uncover an attribute graph. We show that one can fit an effective graphical model that models a structural equation model between latent codes. Using a pre-trained generative autoencoder trained on a large dataset of small molecules, we demonstrate that the graphical model can be used to predict a specific property.
  arXiv  Detail & Related papers  (2022-07-14T19:20:30Z)
• Few-Shot Diffusion Models [15.828257653106537]
  We present Few-Shot Diffusion Models (FSDM), a framework for few-shot generation leveraging conditional DDPMs. FSDM is trained to adapt the generative process conditioned on a small set of images from a given class by aggregating image patch information. We empirically show that FSDM can perform few-shot generation and transfer to new datasets.
  arXiv  Detail & Related papers  (2022-05-30T23:20:33Z)
• Mix and Match: Learning-free Controllable Text Generation using Energy Language Models [33.97800741890231]
  We propose Mix and Match LM, a global score-based alternative for controllable text generation. We interpret the task of controllable generation as drawing samples from an energy-based model. We use a Metropolis-Hastings sampling scheme to sample from this energy-based model.
  arXiv  Detail & Related papers  (2022-03-24T18:52:09Z)
• A Multi-attribute Controllable Generative Model for Histopathology Image Synthesis [13.353006960284159]
  We build upon state-of-the-art generative adversarial networks (GANs) to achieve an attribute-aware image synthesis model, termed AttributeGAN. We conduct experiments on a histopathology dataset containing stained H&E images of urothelial carcinoma.
  arXiv  Detail & Related papers  (2021-11-10T22:48:55Z)
• Controllable and Compositional Generation with Latent-Space Energy-Based Models [60.87740144816278]
  Controllable generation is one of the key requirements for successful adoption of deep generative models in real-world applications. In this work, we use energy-based models (EBMs) to handle compositional generation over a set of attributes. By composing energy functions with logical operators, this work is the first to achieve such compositionality in generating photo-realistic images of resolution 1024x1024.
  arXiv  Detail & Related papers  (2021-10-21T03:31:45Z)
• Flow Plugin Network for conditional generation [1.123376893295777]
  by default, we cannot control its sampling process, i.e., we cannot generate a sample with a specific set of attributes. We propose a novel approach that enables to a generation of objects with a given set of attributes without retraining the base model.
  arXiv  Detail & Related papers  (2021-10-07T17:26:57Z)
• Multilinear Latent Conditioning for Generating Unseen Attribute Combinations [61.686839319971334]
  We show that variational autoencoders (VAE) and generative adversarial networks (GAN) lack the generalization ability that occurs naturally in human perception. We introduce a multilinear latent conditioning framework that captures the multiplicative interactions between attributes. Altogether, we design a novel conditioning framework that can be used with any architecture to synthesize unseen attribute combinations.
  arXiv  Detail & Related papers  (2020-09-09T02:23:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.