Evolution of Activation Functions for Deep Learning-Based Image Classification

• URL: http://arxiv.org/abs/2206.12089v1
• Date: Fri, 24 Jun 2022 05:58:23 GMT
• Title: Evolution of Activation Functions for Deep Learning-Based Image Classification
• Authors: Raz Lapid and Moshe Sipper
• Abstract summary: Activation functions (AFs) play a pivotal role in the performance of neural networks. We propose a novel, three-population, coevolutionary algorithm to evolve AFs. Tested on four datasets -- MNIST, FashionMNIST, KMNIST, and USPS -- coevolution proves to be a performant algorithm for finding good AFs and AF architectures.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, coevolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets -- MNIST, FashionMNIST, KMNIST, and USPS -- coevolution proves to be a performant algorithm for finding good AFs and AF architectures.

Related papers

• Optimizing cnn-Bigru performance: Mish activation and comparative analysis with Relu [0.0]
  Activation functions (AF) are fundamental components within neural networks, enabling them to capture complex patterns and relationships in the data. This study illuminates the effectiveness of AF in elevating the performance of intrusion detection systems.
  arXiv  Detail & Related papers  (2024-05-30T21:48:56Z)
• DCNN: Dual Cross-current Neural Networks Realized Using An Interactive Deep Learning Discriminator for Fine-grained Objects [48.65846477275723]
  This study proposes novel dual-current neural networks (DCNN) to improve the accuracy of fine-grained image classification. The main novel design features for constructing a weakly supervised learning backbone model DCNN include (a) extracting heterogeneous data, (b) keeping the feature map resolution unchanged, (c) expanding the receptive field, and (d) fusing global representations and local features.
  arXiv  Detail & Related papers  (2024-05-07T07:51:28Z)
• Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks [17.12834153477201]
  We propose a novel resource-efficient supervised feature selection method using sparse neural networks. By gradually pruning the uninformative features from the input layer of a sparse neural network trained from scratch, NeuroFS derives an informative subset of features efficiently. NeuroFS achieves the highest ranking-based score among the considered state-of-the-art supervised feature selection models.
  arXiv  Detail & Related papers  (2023-03-10T17:09:55Z)
• GAAF: Searching Activation Functions for Binary Neural Networks through Genetic Algorithm [15.403807679886716]
  Binary neural networks (BNNs) show promising utilization in cost and power-restricted domains such as edge devices and mobile systems. We propose to add a complementary activation function (AF) ahead of the sign based binarization, and rely on the genetic algorithm (GA) to automatically search for the ideal AFs.
  arXiv  Detail & Related papers  (2022-06-05T06:36:22Z)
• A Comprehensive Survey and Performance Analysis of Activation Functions in Deep Learning [23.83339228535986]
  Various types of neural networks have been introduced to deal with different types of problems. The main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish.
  arXiv  Detail & Related papers  (2021-09-29T16:41:19Z)
• Tensor-based framework for training flexible neural networks [9.176056742068813]
  We propose a new learning algorithm which solves a constrained coupled matrix-tensor factorization (CMTF) problem. The proposed algorithm can handle different bases decomposition. The goal of this method is to compress large pretrained NN models, by replacing tensorworks, em i.e., one or multiple layers of the original network, by a new flexible layer.
  arXiv  Detail & Related papers  (2021-06-25T10:26:48Z)
• Involution: Inverting the Inherence of Convolution for Visual Recognition [72.88582255910835]
  We present a novel atomic operation for deep neural networks by inverting the principles of convolution, coined as involution. The proposed involution operator could be leveraged as fundamental bricks to build the new generation of neural networks for visual recognition. Our involution-based models improve the performance of convolutional baselines using ResNet-50 by up to 1.6% top-1 accuracy, 2.5% and 2.4% bounding box AP, and 4.7% mean IoU absolutely.
  arXiv  Detail & Related papers  (2021-03-10T18:40:46Z)
• Binarizing MobileNet via Evolution-based Searching [66.94247681870125]
  We propose a use of evolutionary search to facilitate the construction and training scheme when binarizing MobileNet. Inspired by one-shot architecture search frameworks, we manipulate the idea of group convolution to design efficient 1-Bit Convolutional Neural Networks (CNNs) Our objective is to come up with a tiny yet efficient binary neural architecture by exploring the best candidates of the group convolution.
  arXiv  Detail & Related papers  (2020-05-13T13:25:51Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)
• MSE-Optimal Neural Network Initialization via Layer Fusion [68.72356718879428]
  Deep neural networks achieve state-of-the-art performance for a range of classification and inference tasks. The use of gradient combined nonvolutionity renders learning susceptible to novel problems. We propose fusing neighboring layers of deeper networks that are trained with random variables.
  arXiv  Detail & Related papers  (2020-01-28T18:25:15Z)
• Alpha Discovery Neural Network based on Prior Knowledge [55.65102700986668]
  Genetic programming (GP) is the state-of-the-art in financial automated feature construction task. This paper proposes Alpha Discovery Neural Network (ADNN), a tailored neural network structure which can automatically construct diversified financial technical indicators.
  arXiv  Detail & Related papers  (2019-12-26T03:10:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.