First Power Linear Unit with Sign

• URL: http://arxiv.org/abs/2111.14349v1
• Date: Mon, 29 Nov 2021 06:47:58 GMT
• Title: First Power Linear Unit with Sign
• Authors: Boxi Duan
• Abstract summary: It is enlightened by common inverse operation while endowed with an intuitive meaning of bionics. We extend the function presented to a more generalized type called PFPLUS with two parameters that can be fixed or learnable.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes a novel and insightful activation method termed FPLUS, which exploits mathematical power function with polar signs in form. It is enlightened by common inverse operation while endowed with an intuitive meaning of bionics. The formulation is derived theoretically under conditions of some prior knowledge and anticipative properties, and then its feasibility is verified through a series of experiments using typical benchmark datasets, whose results indicate our approach owns superior competitiveness among numerous activation functions, as well as compatible stability across many CNN architectures. Furthermore, we extend the function presented to a more generalized type called PFPLUS with two parameters that can be fixed or learnable, so as to augment its expressive capacity, and outcomes of identical tests validate this improvement.

Related papers

• Tripod: Three Complementary Inductive Biases for Disentangled Representation Learning [52.70210390424605]
  In this work, we consider endowing a neural network autoencoder with three select inductive biases from the literature. In practice, however, naively combining existing techniques instantiating these inductive biases fails to yield significant benefits. We propose adaptations to the three techniques that simplify the learning problem, equip key regularization terms with stabilizing invariances, and quash degenerate incentives. The resulting model, Tripod, achieves state-of-the-art results on a suite of four image disentanglement benchmarks.
  arXiv  Detail & Related papers  (2024-04-16T04:52:41Z)
• EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention [88.45459681677369]
  We propose a novel transformer variant with complex vector attention, named EulerFormer. It provides a unified theoretical framework to formulate both semantic difference and positional difference. It is more robust to semantic variations and possesses moresuperior theoretical properties in principle.
  arXiv  Detail & Related papers  (2024-03-26T14:18:43Z)
• Mutual Exclusivity Training and Primitive Augmentation to Induce Compositionality [84.94877848357896]
  Recent datasets expose the lack of the systematic generalization ability in standard sequence-to-sequence models. We analyze this behavior of seq2seq models and identify two contributing factors: a lack of mutual exclusivity bias and the tendency to memorize whole examples. We show substantial empirical improvements using standard sequence-to-sequence models on two widely-used compositionality datasets.
  arXiv  Detail & Related papers  (2022-11-28T17:36:41Z)
• A Flexible Approach for Normal Approximation of Geometric and Topological Statistics [8.658596218544774]
  We derive normal approximation results for a class of stabilizing functionals of binomial or Poisson point process. We combine this flexible notion with the theory of strong stabilization to establish our results.
  arXiv  Detail & Related papers  (2022-10-19T17:36:50Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• BEAUTY Powered BEAST [5.423841802973231]
  We study distribution-free goodness-of-fit tests with the proposed Binary Expansion Approximation of UniformiTY (BEAUTY) approach. This theory enables a unification of many important tests of independence via approximations from specific forms of symmetry statistics. To achieve a robust power, we examine test statistics with data-adaptive weights, referred to as the Binary Expansion Adaptive Symmetry Test (BEAST)
  arXiv  Detail & Related papers  (2021-03-01T00:36:15Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• An Investigation of Potential Function Designs for Neural CRF [75.79555356970344]
  In this paper, we investigate a series of increasingly expressive potential functions for neural CRF models. Our experiments show that the decomposed quadrilinear potential function based on the vector representations of two neighboring labels and two neighboring words consistently achieves the best performance.
  arXiv  Detail & Related papers  (2020-11-11T07:32:18Z)
• Squashing activation functions in benchmark tests: towards eXplainable Artificial Intelligence using continuous-valued logic [0.0]
  This work demonstrates the first benchmark tests that measure the performance of Squashing functions in neural networks. Three experiments were carried out to examine their usability and a comparison with the most popular activation functions was made for five different network types. Results indicate that due to the embedded nilpotent logical operators and the differentiability of the Squashing function, it is possible to solve classification problems.
  arXiv  Detail & Related papers  (2020-10-17T10:42:40Z)
• AMITE: A Novel Polynomial Expansion for Analyzing Neural Network Nonlinearities [1.8761314918771685]
  Polynomial expansions are important in the analysis of neural network nonlinearities. Existing approaches span classical Taylor and Chebyshev methods. There are no approaches that provide a consistent method an expansion with all these properties.
  arXiv  Detail & Related papers  (2020-07-13T07:58:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.