RepControlNet: ControlNet Reparameterization

• URL: http://arxiv.org/abs/2408.09240v1
• Date: Sat, 17 Aug 2024 16:21:51 GMT
• Title: RepControlNet: ControlNet Reparameterization
• Authors: Zhaoli Deng, Kaibin Zhou, Fanyi Wang, Zhenpeng Mi,
• Abstract summary: RepControlNet is proposed to realize controllable generation of diffusion models without increasing computation. We have carried out a large number of experiments on both SD1.5 and SDXL, and the experimental results show the effectiveness and efficiency of the proposed RepControlNet.
• Score: 0.562479170374811
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: With the wide application of diffusion model, the high cost of inference resources has became an important bottleneck for its universal application. Controllable generation, such as ControlNet, is one of the key research directions of diffusion model, and the research related to inference acceleration and model compression is more important. In order to solve this problem, this paper proposes a modal reparameterization method, RepControlNet, to realize the controllable generation of diffusion models without increasing computation. In the training process, RepControlNet uses the adapter to modulate the modal information into the feature space, copy the CNN and MLP learnable layers of the original diffusion model as the modal network, and initialize these weights based on the original weights and coefficients. The training process only optimizes the parameters of the modal network. In the inference process, the weights of the neutralization original diffusion model in the modal network are reparameterized, which can be compared with or even surpass the methods such as ControlNet, which use additional parameters and computational quantities, without increasing the number of parameters. We have carried out a large number of experiments on both SD1.5 and SDXL, and the experimental results show the effectiveness and efficiency of the proposed RepControlNet.

Related papers

• RCDM: Enabling Robustness for Conditional Diffusion Model [2.4915590770454035]
  Conditional diffusion model (CDM) enhances the standard diffusion model by providing more control. Inaccurate conditional inputs in the inverse process of CDM can easily lead to generating fixed errors in the neural network. We propose a lightweight solution, the Robust Conditional Diffusion Model (RCDM)
  arXiv  Detail & Related papers  (2024-08-05T13:12:57Z)
• Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review [63.31328039424469]
  This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
  arXiv  Detail & Related papers  (2024-07-18T17:35:32Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Do deep neural networks utilize the weight space efficiently? [2.9914612342004503]
  Deep learning models like Transformers and Convolutional Neural Networks (CNNs) have revolutionized various domains, but their parameter-intensive nature hampers deployment in resource-constrained settings. We introduce a novel concept utilizing column space and row space of weight matrices, which allows for a substantial reduction in model parameters without compromising performance. Our approach applies to both Bottleneck and Attention layers, effectively halving the parameters while incurring only minor performance degradation.
  arXiv  Detail & Related papers  (2024-01-26T21:51:49Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Rewarded meta-pruning: Meta Learning with Rewards for Channel Pruning [19.978542231976636]
  This paper proposes a novel method to reduce the parameters and FLOPs for computational efficiency in deep learning models. We introduce accuracy and efficiency coefficients to control the trade-off between the accuracy of the network and its computing efficiency.
  arXiv  Detail & Related papers  (2023-01-26T12:32:01Z)
• Deep Hierarchy Quantization Compression algorithm based on Dynamic Sampling [11.439540966972212]
  Federated machine learning stores data locally for training and aggregates the models on the server. During the training process, the transmission of model parameters can impose a significant load on the network bandwidth. We propose a deep hierarchical quantization compression algorithm, which further compresses the model and reduces the network load brought by data transmission.
  arXiv  Detail & Related papers  (2022-12-30T15:12:30Z)
• Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
  We construct a dataset of neural network checkpoints and train a generative model on the parameters. We find that our approach successfully generates parameters for a wide range of loss prompts. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2022-09-26T17:59:58Z)
• MoEfication: Conditional Computation of Transformer Models for Efficient Inference [66.56994436947441]
  Transformer-based pre-trained language models can achieve superior performance on most NLP tasks due to large parameter capacity, but also lead to huge computation cost. We explore to accelerate large-model inference by conditional computation based on the sparse activation phenomenon. We propose to transform a large model into its mixture-of-experts (MoE) version with equal model size, namely MoEfication.
  arXiv  Detail & Related papers  (2021-10-05T02:14:38Z)
• Model Fusion via Optimal Transport [64.13185244219353]
  We present a layer-wise model fusion algorithm for neural networks. We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
  arXiv  Detail & Related papers  (2019-10-12T22:07:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.