PowerLinear Activation Functions with application to the first layer of CNNs

• URL: http://arxiv.org/abs/2108.09256v1
• Date: Fri, 20 Aug 2021 16:43:01 GMT
• Title: PowerLinear Activation Functions with application to the first layer of CNNs
• Authors: Kamyar Nasiri, Kamaledin Ghiasi-Shirazi
• Abstract summary: EvenPowLin activation functions are used in CNN models to classify the inversion of grayscale images. EvenPowLin activation functions are used in CNN models to classify the inversion of grayscale images as accurately as the original grayscale images.
• Score: 0.609170287691728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Convolutional neural networks (CNNs) have become the state-of-the-art tool for dealing with unsolved problems in computer vision and image processing. Since the convolution operator is a linear operator, several generalizations have been proposed to improve the performance of CNNs. One way to increase the capability of the convolution operator is by applying activation functions on the inner product operator. In this paper, we will introduce PowerLinear activation functions, which are based on the polynomial kernel generalization of the convolution operator. EvenPowLin functions are the main branch of the PowerLinear activation functions. This class of activation functions is saturated neither in the positive input region nor in the negative one. Also, the negative inputs are activated with the same magnitude as the positive inputs. These features made the EvenPowLin activation functions able to be utilized in the first layer of CNN architectures and learn complex features of input images. Additionally, EvenPowLin activation functions are used in CNN models to classify the inversion of grayscale images as accurately as the original grayscale images, which is significantly better than commonly used activation functions.

Related papers

• Kolmogorov-Arnold Transformer [72.88137795439407]
  We introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces layers with Kolmogorov-Arnold Network (KAN) layers. We identify three key challenges: (C1) Base function, (C2) Inefficiency, and (C3) Weight. With these designs, KAT outperforms traditional-based transformers.
  arXiv  Detail & Related papers  (2024-09-16T17:54:51Z)
• Trainable Highly-expressive Activation Functions [8.662179223772089]
  We introduce DiTAC, a trainable highly-expressive activation function. DiTAC enhances model expressiveness and performance, often yielding substantial improvements. It also outperforms existing activation functions (regardless of whether the latter are fixed or trainable) in tasks such as semantic segmentation, image generation, regression problems, and image classification.
  arXiv  Detail & Related papers  (2024-07-10T11:49:29Z)
• Multilinear Operator Networks [60.7432588386185]
  Polynomial Networks is a class of models that does not require activation functions. We propose MONet, which relies solely on multilinear operators.
  arXiv  Detail & Related papers  (2024-01-31T16:52:19Z)
• STL: A Signed and Truncated Logarithm Activation Function for Neural Networks [5.9622541907827875]
  Activation functions play an essential role in neural networks. We present a novel signed and truncated logarithm function as activation function. The suggested activation function can be applied in a large range of neural networks.
  arXiv  Detail & Related papers  (2023-07-31T03:41:14Z)
• Evaluating CNN with Oscillatory Activation Function [0.0]
  CNNs capability to learn high-dimensional complex features from the images is the non-linearity introduced by the activation function. This paper explores the performance of one of the CNN architecture ALexNet on MNIST and CIFAR10 datasets using oscillating activation function (GCU) and some other commonly used activation functions like ReLu, PReLu, and Mish.
  arXiv  Detail & Related papers  (2022-11-13T11:17:13Z)
• Provable General Function Class Representation Learning in Multitask Bandits and MDPs [58.624124220900306]
  multitask representation learning is a popular approach in reinforcement learning to boost the sample efficiency. In this work, we extend the analysis to general function class representations. We theoretically validate the benefit of multitask representation learning within general function class for bandits and linear MDP.
  arXiv  Detail & Related papers  (2022-05-31T11:36:42Z)
• Graph-adaptive Rectified Linear Unit for Graph Neural Networks [64.92221119723048]
  Graph Neural Networks (GNNs) have achieved remarkable success by extending traditional convolution to learning on non-Euclidean data. We propose Graph-adaptive Rectified Linear Unit (GReLU) which is a new parametric activation function incorporating the neighborhood information in a novel and efficient way. We conduct comprehensive experiments to show that our plug-and-play GReLU method is efficient and effective given different GNN backbones and various downstream tasks.
  arXiv  Detail & Related papers  (2022-02-13T10:54:59Z)
• Neural networks with trainable matrix activation functions [7.999703756441757]
  This work develops a systematic approach to constructing matrix-valued activation functions. The proposed activation functions depend on parameters that are trained along with the weights and bias vectors.
  arXiv  Detail & Related papers  (2021-09-21T04:11:26Z)
• Content-Aware Convolutional Neural Networks [98.97634685964819]
  Convolutional Neural Networks (CNNs) have achieved great success due to the powerful feature learning ability of convolution layers. We propose a Content-aware Convolution (CAC) that automatically detects the smooth windows and applies a 1x1 convolutional kernel to replace the original large kernel.
  arXiv  Detail & Related papers  (2021-06-30T03:54:35Z)
• Deep Polynomial Neural Networks [77.70761658507507]
  $Pi$Nets are a new class of function approximators based on expansions. $Pi$Nets produce state-the-art results in three challenging tasks, i.e. image generation, face verification and 3D mesh representation learning.
  arXiv  Detail & Related papers  (2020-06-20T16:23:32Z)
• Trainable Activation Function in Image Classification [0.0]
  This paper focus on how to make the activation function trainable for deep neural networks. We use series and linear combination of different activation functions make activation functions continuously variable.
  arXiv  Detail & Related papers  (2020-04-28T03:50:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.