Meta ControlNet: Enhancing Task Adaptation via Meta Learning

• URL: http://arxiv.org/abs/2312.01255v1
• Date: Sun, 3 Dec 2023 01:36:45 GMT
• Title: Meta ControlNet: Enhancing Task Adaptation via Meta Learning
• Authors: Junjie Yang, Jinze Zhao, Peihao Wang, Zhangyang Wang, Yingbin Liang
• Abstract summary: Diffusion-based image synthesis has attracted extensive attention recently. ControlNet that uses image-based prompts exhibits powerful capability in image tasks such as canny edge detection and generates images well aligned with these prompts. Recent context-learning approaches have improved its adaptability, but mainly for edge-based tasks, and rely on paired examples. We introduce a novel Meta ControlNet method, which adopts the task-agnostic meta learning technique and features a new layer freezing design.
• Score: 111.16980465577682
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion-based image synthesis has attracted extensive attention recently. In particular, ControlNet that uses image-based prompts exhibits powerful capability in image tasks such as canny edge detection and generates images well aligned with these prompts. However, vanilla ControlNet generally requires extensive training of around 5000 steps to achieve a desirable control for a single task. Recent context-learning approaches have improved its adaptability, but mainly for edge-based tasks, and rely on paired examples. Thus, two important open issues are yet to be addressed to reach the full potential of ControlNet: (i) zero-shot control for certain tasks and (ii) faster adaptation for non-edge-based tasks. In this paper, we introduce a novel Meta ControlNet method, which adopts the task-agnostic meta learning technique and features a new layer freezing design. Meta ControlNet significantly reduces learning steps to attain control ability from 5000 to 1000. Further, Meta ControlNet exhibits direct zero-shot adaptability in edge-based tasks without any finetuning, and achieves control within only 100 finetuning steps in more complex non-edge tasks such as Human Pose, outperforming all existing methods. The codes is available in https://github.com/JunjieYang97/Meta-ControlNet.

Related papers

• CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation [69.43106794519193]
  We propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions. Our framework reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights.
  arXiv  Detail & Related papers  (2024-10-12T07:04:32Z)
• Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion [27.61734719689046]
  We propose a training-free approach named Mask-guided Prompt Following (MGPF) to enhance prompt following with visual control. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments.
  arXiv  Detail & Related papers  (2024-04-23T06:10:43Z)
• Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Model [62.51232333352754]
  Ctrl-Adapter adds diverse controls to any image/video diffusion model through the adaptation of pretrained ControlNets. With six diverse U-Net/DiT-based image/video diffusion models, Ctrl-Adapter matches the performance of pretrained ControlNets on COCO.
  arXiv  Detail & Related papers  (2024-04-15T17:45:36Z)
• FineControlNet: Fine-level Text Control for Image Generation with Spatially Aligned Text Control Injection [28.65209293141492]
  FineControlNet provides fine control over each instance's appearance while maintaining the precise pose control capability. FineControlNet achieves superior performance in generating images that follow the user-provided instance-specific text prompts and poses.
  arXiv  Detail & Related papers  (2023-12-14T18:59:43Z)
• CCM: Adding Conditional Controls to Text-to-Image Consistency Models [89.75377958996305]
  We consider alternative strategies for adding ControlNet-like conditional control to Consistency Models. A lightweight adapter can be jointly optimized under multiple conditions through Consistency Training. We study these three solutions across various conditional controls, including edge, depth, human pose, low-resolution image and masked image.
  arXiv  Detail & Related papers  (2023-12-12T04:16:03Z)
• Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Models [82.19740045010435]
  We introduce Uni-ControlNet, a unified framework that allows for the simultaneous utilization of different local controls and global controls. Unlike existing methods, Uni-ControlNet only requires the fine-tuning of two additional adapters upon frozen pre-trained text-to-image diffusion models. Uni-ControlNet demonstrates its superiority over existing methods in terms of controllability, generation quality and composability.
  arXiv  Detail & Related papers  (2023-05-25T17:59:58Z)
• Meta-Learning Online Control for Linear Dynamical Systems [2.867517731896504]
  We propose a meta-learning online control algorithm for the control setting. We characterize its performance by textitmeta-regret, the average cumulative regret across the tasks. We show that when the number of tasks are sufficiently large, our proposed approach achieves a meta-regret that is smaller by a factor $D/D*$ compared to an independent-learning online control algorithm.
  arXiv  Detail & Related papers  (2022-08-18T20:44:07Z)
• Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
  Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers. Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters. We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
  arXiv  Detail & Related papers  (2022-03-30T23:16:07Z)
• A Meta-Reinforcement Learning Approach to Process Control [3.9146761527401424]
  Meta-learning aims to quickly adapt models, such as neural networks, to perform new tasks. We construct a controller and meta-train the controller using a latent context variable through a separate embedding neural network. In both cases, our meta-learning algorithm adapts very quickly to new tasks, outperforming a regular DRL controller trained from scratch.
  arXiv  Detail & Related papers  (2021-03-25T18:20:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.