CFTS-GAN: Continual Few-Shot Teacher Student for Generative Adversarial Networks

• URL: http://arxiv.org/abs/2410.14749v1
• Date: Thu, 17 Oct 2024 20:49:08 GMT
• Title: CFTS-GAN: Continual Few-Shot Teacher Student for Generative Adversarial Networks
• Authors: Munsif Ali, Leonardo Rossi, Massimo Bertozzi,
• Abstract summary: Few-shot and continual learning face two well-known challenges in GANs: overfitting and catastrophic forgetting. This paper proposes a Continual Few-shot Teacher-Student technique for the generative adversarial network (CFTS-GAN) that considers both challenges together.
• Score: 0.5024983453990064
• License:
• Abstract: Few-shot and continual learning face two well-known challenges in GANs: overfitting and catastrophic forgetting. Learning new tasks results in catastrophic forgetting in deep learning models. In the case of a few-shot setting, the model learns from a very limited number of samples (e.g. 10 samples), which can lead to overfitting and mode collapse. So, this paper proposes a Continual Few-shot Teacher-Student technique for the generative adversarial network (CFTS-GAN) that considers both challenges together. Our CFTS-GAN uses an adapter module as a student to learn a new task without affecting the previous knowledge. To make the student model efficient in learning new tasks, the knowledge from a teacher model is distilled to the student. In addition, the Cross-Domain Correspondence (CDC) loss is used by both teacher and student to promote diversity and to avoid mode collapse. Moreover, an effective strategy of freezing the discriminator is also utilized for enhancing performance. Qualitative and quantitative results demonstrate more diverse image synthesis and produce qualitative samples comparatively good to very stronger state-of-the-art models.

Related papers

• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Toward In-Context Teaching: Adapting Examples to Students' Misconceptions [54.82965010592045]
  We introduce a suite of models and evaluation methods we call AdapT. AToM is a new probabilistic model for adaptive teaching that jointly infers students' past beliefs and optimize for the correctness of future beliefs. Our results highlight both the difficulty of the adaptive teaching task and the potential of learned adaptive models for solving it.
  arXiv  Detail & Related papers  (2024-05-07T17:05:27Z)
• L2T-DLN: Learning to Teach with Dynamic Loss Network [4.243592852049963]
  In existing works, the teacher iteration model 1) merely determines the loss function based on the present states of the student model. In this paper, we first formulate the loss adjustment as a temporal task by designing a teacher model with memory units. Then, with a dynamic loss network, we can additionally use the states of the loss to assist the teacher learning in enhancing the interactions between the teacher and the student model.
  arXiv  Detail & Related papers  (2023-10-30T07:21:40Z)
• Common Knowledge Learning for Generating Transferable Adversarial Examples [60.1287733223249]
  This paper focuses on an important type of black-box attacks, where the adversary generates adversarial examples by a substitute (source) model. Existing methods tend to give unsatisfactory adversarial transferability when the source and target models are from different types of DNN architectures. We propose a common knowledge learning (CKL) framework to learn better network weights to generate adversarial examples.
  arXiv  Detail & Related papers  (2023-07-01T09:07:12Z)
• End to End Generative Meta Curriculum Learning For Medical Data Augmentation [2.471925498075058]
  Current medical image synthetic augmentation techniques rely on intensive use of generative adversarial networks (GANs) We introduce a novel generative meta curriculum learning method that trains the task-specific model (student) end-to-end with only one additional teacher model. In contrast to the generator and discriminator in GAN, which compete with each other, the teacher and student collaborate to improve the student's performance on the target tasks.
  arXiv  Detail & Related papers  (2022-12-20T08:54:11Z)
• Distantly-Supervised Named Entity Recognition with Adaptive Teacher Learning and Fine-grained Student Ensemble [56.705249154629264]
  Self-training teacher-student frameworks are proposed to improve the robustness of NER models. In this paper, we propose an adaptive teacher learning comprised of two teacher-student networks. Fine-grained student ensemble updates each fragment of the teacher model with a temporal moving average of the corresponding fragment of the student, which enhances consistent predictions on each model fragment against noise.
  arXiv  Detail & Related papers  (2022-12-13T12:14:09Z)
• Dynamic Contrastive Distillation for Image-Text Retrieval [90.05345397400144]
  We present a novel plug-in dynamic contrastive distillation (DCD) framework to compress image-text retrieval models. We successfully apply our proposed DCD strategy to two state-of-the-art vision-language pretrained models, i.e. ViLT and METER. Experiments on MS-COCO and Flickr30K benchmarks show the effectiveness and efficiency of our DCD framework.
  arXiv  Detail & Related papers  (2022-07-04T14:08:59Z)
• Revisiting Knowledge Distillation: An Inheritance and Exploration Framework [153.73692961660964]
  Knowledge Distillation (KD) is a popular technique to transfer knowledge from a teacher model to a student model. We propose a novel inheritance and exploration knowledge distillation framework (IE-KD) Our IE-KD framework is generic and can be easily combined with existing distillation or mutual learning methods for training deep neural networks.
  arXiv  Detail & Related papers  (2021-07-01T02:20:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.