Activation Function Optimization Scheme for Image Classification

• URL: http://arxiv.org/abs/2409.04915v1
• Date: Sat, 7 Sep 2024 21:40:15 GMT
• Title: Activation Function Optimization Scheme for Image Classification
• Authors: Abdur Rahman, Lu He, Haifeng Wang,
• Abstract summary: Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. We propose an evolutionary approach for optimizing activation functions specifically for image classification tasks. We obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU)
• Score: 20.531966402727708
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during deep learning model development. Existing state-of-the-art activation functions are manually designed with human expertise except for Swish. Swish was developed using a reinforcement learning-based search strategy. In this study, we propose an evolutionary approach for optimizing activation functions specifically for image classification tasks, aiming to discover functions that outperform current state-of-the-art options. Through this optimization framework, we obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU). The developed activation functions are evaluated for image classification tasks from two perspectives: (1) five state-of-the-art neural network architectures, such as ResNet50, AlexNet, VGG16, MobileNet, and Compact Convolutional Transformer which cover computationally heavy to light neural networks, and (2) eight standard datasets, including CIFAR10, Imagenette, MNIST, Fashion MNIST, Beans, Colorectal Histology, CottonWeedID15, and TinyImageNet which cover from typical machine vision benchmark, agricultural image applications to medical image applications. Finally, we statistically investigate the generalization of the resultant activation functions developed through the optimization scheme. With a Friedman test, we conclude that the optimization scheme is able to generate activation functions that outperform the existing standard ones in 92.8% cases among 28 different cases studied, and $-x\cdot erf(e^{-x})$ is found to be the best activation function for image classification generated by the optimization scheme.

Related papers

• LeRF: Learning Resampling Function for Adaptive and Efficient Image Interpolation [64.34935748707673]
  Recent deep neural networks (DNNs) have made impressive progress in performance by introducing learned data priors. We propose a novel method of Learning Resampling (termed LeRF) which takes advantage of both the structural priors learned by DNNs and the locally continuous assumption. LeRF assigns spatially varying resampling functions to input image pixels and learns to predict the shapes of these resampling functions with a neural network.
  arXiv  Detail & Related papers  (2024-07-13T16:09:45Z)
• Multilinear Operator Networks [60.7432588386185]
  Polynomial Networks is a class of models that does not require activation functions. We propose MONet, which relies solely on multilinear operators.
  arXiv  Detail & Related papers  (2024-01-31T16:52:19Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Efficient Activation Function Optimization through Surrogate Modeling [15.219959721479835]
  This paper aims to improve the state of the art through three steps. First, the benchmark Act-Bench-CNN, Act-Bench-ResNet, and Act-Bench-ViT were created by training convolutional, residual, and vision transformer architectures. Second, a characterization of the benchmark space was developed, leading to a new surrogate-based method for optimization.
  arXiv  Detail & Related papers  (2023-01-13T23:11:14Z)
• Evaluating CNN with Oscillatory Activation Function [0.0]
  CNNs capability to learn high-dimensional complex features from the images is the non-linearity introduced by the activation function. This paper explores the performance of one of the CNN architecture ALexNet on MNIST and CIFAR10 datasets using oscillating activation function (GCU) and some other commonly used activation functions like ReLu, PReLu, and Mish.
  arXiv  Detail & Related papers  (2022-11-13T11:17:13Z)
• Learning specialized activation functions with the Piecewise Linear Unit [7.820667552233989]
  We propose a new activation function called Piecewise Linear Unit(PWLU), which incorporates a carefully designed formulation and learning method. It can learn specialized activation functions and achieves SOTA performance on large-scale datasets like ImageNet and COCO. PWLU is also easy to implement and efficient at inference, which can be widely applied in real-world applications.
  arXiv  Detail & Related papers  (2021-04-08T11:29:11Z)
• Comparisons among different stochastic selection of activation layers for convolutional neural networks for healthcare [77.99636165307996]
  We classify biomedical images using ensembles of neural networks. We select our activations among the following ones: ReLU, leaky ReLU, Parametric ReLU, ELU, Adaptive Piecewice Linear Unit, S-Shaped ReLU, Swish, Mish, Mexican Linear Unit, Parametric Deformable Linear Unit, Soft Root Sign.
  arXiv  Detail & Related papers  (2020-11-24T01:53:39Z)
• Generative Hierarchical Features from Synthesizing Images [65.66756821069124]
  We show that learning to synthesize images can bring remarkable hierarchical visual features that are generalizable across a wide range of applications. The visual feature produced by our encoder, termed as Generative Hierarchical Feature (GH-Feat), has strong transferability to both generative and discriminative tasks.
  arXiv  Detail & Related papers  (2020-07-20T18:04:14Z)
• Discovering Parametric Activation Functions [17.369163074697475]
  This paper proposes a technique for customizing activation functions automatically, resulting in reliable improvements in performance. Experiments with four different neural network architectures on the CIFAR-10 and CIFAR-100 image classification datasets show that this approach is effective.
  arXiv  Detail & Related papers  (2020-06-05T00:25:33Z)
• Evolving Normalization-Activation Layers [100.82879448303805]
  We develop efficient rejection protocols to quickly filter out candidate layers that do not work well. Our method leads to the discovery of EvoNorms, a set of new normalization-activation layers with novel, and sometimes surprising structures. Our experiments show that EvoNorms work well on image classification models including ResNets, MobileNets and EfficientNets.
  arXiv  Detail & Related papers  (2020-04-06T19:52:48Z)
• Evolutionary Optimization of Deep Learning Activation Functions [15.628118691027328]
  We show that evolutionary algorithms can discover novel activation functions that outperform the Rectified Linear Unit (ReLU) replacing ReLU with evolved activation functions results in statistically significant increases in network accuracy. These novel activation functions are shown to generalize, achieving high performance across tasks.
  arXiv  Detail & Related papers  (2020-02-17T19:54:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.