Efficient Continual Adaptation for Generative Adversarial Networks

• URL: http://arxiv.org/abs/2103.04032v1
• Date: Sat, 6 Mar 2021 05:09:37 GMT
• Title: Efficient Continual Adaptation for Generative Adversarial Networks
• Authors: Sakshi Varshney, Vinay Kumar Verma, Lawrence Carin, Piyush Rai
• Abstract summary: We present a continual learning approach for generative adversarial networks (GANs) Our approach is based on learning a set of global and task-specific parameters. We show that the feature-map transformation based approach outperforms state-of-the-art continual GANs methods.
• Score: 97.20244383723853
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a continual learning approach for generative adversarial networks (GANs), by designing and leveraging parameter-efficient feature map transformations. Our approach is based on learning a set of global and task-specific parameters. The global parameters are fixed across tasks whereas the task specific parameters act as local adapters for each task, and help in efficiently transforming the previous task's feature map to the new task's feature map. Moreover, we propose an element-wise residual bias in the transformed feature space which highly stabilizes GAN training. In contrast to the recent approaches for continual GANs, we do not rely on memory replay, regularization towards previous tasks' parameters, or expensive weight transformations. Through extensive experiments on challenging and diverse datasets, we show that the feature-map transformation based approach outperforms state-of-the-art continual GANs methods, with substantially fewer parameters, and also generates high-quality samples that can be used in generative replay based continual learning of discriminative tasks.

Related papers

• Task-Aware Harmony Multi-Task Decision Transformer for Offline Reinforcement Learning [70.96345405979179]
  The purpose of offline multi-task reinforcement learning (MTRL) is to develop a unified policy applicable to diverse tasks without the need for online environmental interaction. variations in task content and complexity pose significant challenges in policy formulation. We introduce the Harmony Multi-Task Decision Transformer (HarmoDT), a novel solution designed to identify an optimal harmony subspace of parameters for each task.
  arXiv  Detail & Related papers  (2024-11-02T05:49:14Z)
• Task-Distributionally Robust Data-Free Meta-Learning [99.56612787882334]
  Data-Free Meta-Learning (DFML) aims to efficiently learn new tasks by leveraging multiple pre-trained models without requiring their original training data. For the first time, we reveal two major challenges hindering their practical deployments: Task-Distribution Shift ( TDS) and Task-Distribution Corruption (TDC)
  arXiv  Detail & Related papers  (2023-11-23T15:46:54Z)
• Task Difficulty Aware Parameter Allocation & Regularization for Lifelong Learning [20.177260510548535]
  We propose the Allocation & Regularization (PAR), which adaptively select an appropriate strategy for each task from parameter allocation and regularization based on its learning difficulty. Our method is scalable and significantly reduces the model's redundancy while improving the model's performance.
  arXiv  Detail & Related papers  (2023-04-11T15:38:21Z)
• Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm [21.747744343882392]
  PETL aims at efficiently adapting large models pre-trained on massive data to downstream tasks with limited task-specific data. This paper proposes a novel two-stage paradigm, where the pre-trained model is first aligned to the target distribution. The proposed paradigm achieves state-of-the-art performance on the average accuracy of 19 downstream tasks.
  arXiv  Detail & Related papers  (2023-03-14T13:50:31Z)
• TAG: Task-based Accumulated Gradients for Lifelong learning [21.779858050277475]
  We propose a task-aware system that adapts the learning rate based on the relatedness among tasks. We empirically show that our proposed adaptive learning rate not only accounts for catastrophic forgetting but also allows positive backward transfer.
  arXiv  Detail & Related papers  (2021-05-11T16:10:32Z)
• Piggyback GAN: Efficient Lifelong Learning for Image Conditioned Generation [48.31909975623379]
  We propose a parameter efficient framework, Piggyback GAN, which learns the current task by building a set of convolutional and deconvolutional filters. For the current task, our model achieves high generation quality on par with a standalone model at a lower number of parameters. We validate Piggyback GAN on various image-conditioned generation tasks across different domains.
  arXiv  Detail & Related papers  (2021-04-24T12:09:52Z)
• Efficient Feature Transformations for Discriminative and Generative Continual Learning [98.10425163678082]
  We propose a simple task-specific feature map transformation strategy for continual learning. Theses provide powerful flexibility for learning new tasks, achieved with minimal parameters added to the base architecture. We demonstrate the efficacy and efficiency of our method with an extensive set of experiments in discriminative (CIFAR-100 and ImageNet-1K) and generative sequences of tasks.
  arXiv  Detail & Related papers  (2021-03-25T01:48:14Z)
• Parameter-Efficient Transfer Learning with Diff Pruning [108.03864629388404]
  diff pruning is a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark.
  arXiv  Detail & Related papers  (2020-12-14T12:34:01Z)
• Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Tasks [30.075430694663293]
  We study the transfer learning process between two linear regression problems. We examine a parameter transfer mechanism whereby a subset of the parameters of the target task solution are constrained to the values learned for a related source task.
  arXiv  Detail & Related papers  (2020-06-12T08:42:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.