Signal Propagation in a Gradient-Based and Evolutionary Learning System

• URL: http://arxiv.org/abs/2102.08929v1
• Date: Wed, 10 Feb 2021 16:46:44 GMT
• Title: Signal Propagation in a Gradient-Based and Evolutionary Learning System
• Authors: Jamal Toutouh and Una-May O'Reilly
• Abstract summary: Coevolutionary algorithms (CEAs) for GAN training are empirically robust to them. We propose Lipi-Ring, a distributed CEA like Lipizzaner, except that it uses a different spatial topology. Our central question is whether the different directionality of signal propagation meets or exceeds the performance quality and training efficiency of Lipizzaner.
• Score: 9.911708222650825
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative adversarial networks (GANs) exhibit training pathologies that can lead to convergence-related degenerative behaviors, whereas spatially-distributed, coevolutionary algorithms (CEAs) for GAN training, e.g. Lipizzaner, are empirically robust to them. The robustness arises from diversity that occurs by training populations of generators and discriminators in each cell of a toroidal grid. Communication, where signals in the form of parameters of the best GAN in a cell propagate in four directions: North, South, West, and East, also plays a role, by communicating adaptations that are both new and fit. We propose Lipi-Ring, a distributed CEA like Lipizzaner, except that it uses a different spatial topology, i.e. a ring. Our central question is whether the different directionality of signal propagation (effectively migration to one or more neighbors on each side of a cell) meets or exceeds the performance quality and training efficiency of Lipizzaner Experimental analysis on different datasets (i.e, MNIST, CelebA, and COVID-19 chest X-ray images) shows that there are no significant differences between the performances of the trained generative models by both methods. However, Lipi-Ring significantly reduces the computational time (14.2%. . . 41.2%). Thus, Lipi-Ring offers an alternative to Lipizzaner when the computational cost of training matters.

Related papers

• Affine-Consistent Transformer for Multi-Class Cell Nuclei Detection [76.11864242047074]
  We propose a novel Affine-Consistent Transformer (AC-Former), which directly yields a sequence of nucleus positions. We introduce an Adaptive Affine Transformer (AAT) module, which can automatically learn the key spatial transformations to warp original images for local network training. Experimental results demonstrate that the proposed method significantly outperforms existing state-of-the-art algorithms on various benchmarks.
  arXiv  Detail & Related papers  (2023-10-22T02:27:02Z)
• Lottery Tickets in Evolutionary Optimization: On Sparse Backpropagation-Free Trainability [0.0]
  We study gradient descent (GD)-based sparse training and evolution strategies (ES) We find that ES explore diverse and flat local optima and do not preserve linear mode connectivity across sparsity levels and independent runs.
  arXiv  Detail & Related papers  (2023-05-31T15:58:54Z)
• Dissecting adaptive methods in GANs [46.90376306847234]
  We study how adaptive methods help train generative adversarial networks (GANs) By considering an update rule with the magnitude of the Adam update and the normalized direction of SGD, we empirically show that the adaptive magnitude of Adam is key for GAN training. We prove that in that setting, GANs trained with nSGDA recover all the modes of the true distribution, whereas the same networks trained with SGDA (and any learning rate configuration) suffer from mode collapse.
  arXiv  Detail & Related papers  (2022-10-09T19:00:07Z)
• Improving GANs with A Dynamic Discriminator [106.54552336711997]
  We argue that a discriminator with an on-the-fly adjustment on its capacity can better accommodate such a time-varying task. A comprehensive empirical study confirms that the proposed training strategy, termed as DynamicD, improves the synthesis performance without incurring any additional cost or training objectives.
  arXiv  Detail & Related papers  (2022-09-20T17:57:33Z)
• Deceive D: Adaptive Pseudo Augmentation for GAN Training with Limited Data [125.7135706352493]
  Generative adversarial networks (GANs) typically require ample data for training in order to synthesize high-fidelity images. Recent studies have shown that training GANs with limited data remains formidable due to discriminator overfitting. This paper introduces a novel strategy called Adaptive Pseudo Augmentation (APA) to encourage healthy competition between the generator and the discriminator.
  arXiv  Detail & Related papers  (2021-11-12T18:13:45Z)
• Local AdaGrad-Type Algorithm for Stochastic Convex-Concave Minimax Problems [80.46370778277186]
  Large scale convex-concave minimax problems arise in numerous applications, including game theory, robust training, and training of generative adversarial networks. We develop a communication-efficient distributed extragrad algorithm, LocalAdaSient, with an adaptive learning rate suitable for solving convex-concave minimax problem in the. Server model. We demonstrate its efficacy through several experiments in both the homogeneous and heterogeneous settings.
  arXiv  Detail & Related papers  (2021-06-18T09:42:05Z)
• Epigenetic evolution of deep convolutional models [81.21462458089142]
  We build upon a previously proposed neuroevolution framework to evolve deep convolutional models. We propose a convolutional layer layout which allows kernels of different shapes and sizes to coexist within the same layer. The proposed layout enables the size and shape of individual kernels within a convolutional layer to be evolved with a corresponding new mutation operator.
  arXiv  Detail & Related papers  (2021-04-12T12:45:16Z)
• Training Generative Adversarial Networks in One Stage [58.983325666852856]
  We introduce a general training scheme that enables training GANs efficiently in only one stage. We show that the proposed method is readily applicable to other adversarial-training scenarios, such as data-free knowledge distillation.
  arXiv  Detail & Related papers  (2021-02-28T09:03:39Z)
• Analyzing the Components of Distributed Coevolutionary GAN Training [8.198369743955528]
  We investigate the impact on the performance of two algorithm components that influence the diversity during coevolution. In experiments on MNIST dataset, we find that the combination of these two components provides the best generative models.
  arXiv  Detail & Related papers  (2020-08-03T18:35:06Z)
• Parallel/distributed implementation of cellular training for generative adversarial neural networks [7.504722086511921]
  Generative adversarial networks (GANs) are widely used to learn generative models. This article presents a parallel/distributed implementation of a cellular competitive coevolutionary method to train two populations of GANs.
  arXiv  Detail & Related papers  (2020-04-07T16:01:58Z)
• Imbalanced Data Learning by Minority Class Augmentation using Capsule Adversarial Networks [31.073558420480964]
  We propose a method to restore the balance in imbalanced images, by coalescing two concurrent methods. In our model, generative and discriminative networks play a novel competitive game. The coalescing of capsule-GAN is effective at recognizing highly overlapping classes with much fewer parameters compared with the convolutional-GAN.
  arXiv  Detail & Related papers  (2020-04-05T12:36:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.