Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations

• URL: http://arxiv.org/abs/2604.07715v1
• Date: Thu, 09 Apr 2026 01:58:51 GMT
• Title: Mathematical analysis of one-layer neural network with fixed biases, a new activation function and other observations
• Abstract summary: We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases.<n>We rigorously prove the convergence of the learning process with the $L2$ squared loss function and the gradient descent procedure.
• Score: 0.8594140167290097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We analyze a simple one-hidden-layer neural network with ReLU activation functions and fixed biases, with one-dimensional input and output. We study both continuous and discrete versions of the model, and we rigorously prove the convergence of the learning process with the $L^2$ squared loss function and the gradient descent procedure. We also prove the spectral bias property for this learning process. Several conclusions of this analysis are discussed; in particular, regarding the structure and properties that activation functions should possess, as well as the relationships between the spectrum of certain operators and the learning process. Based on this, we also propose an alternative activation function, the full-wave rectified exponential function (FReX), and we discuss the convergence of the gradient descent with this alternative activation function.

Related papers

• Neural Approximation and Its Applications [73.11802084858294]
  We introduce the neural basis function by leveraging an untrained neural network as the basis function.<n>We suggest the neural approximation (NeuApprox) paradigm for multivariate function approximation.<n>Experiments on diverse multi-dimensional images, light field data, videos, traffic data, and point cloud data demonstrate the promising performance of NeuApprox.
  arXiv  Detail & Related papers  (2026-03-04T06:05:43Z)
• On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking [49.1352577985191]
  We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task.<n>Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics.
  arXiv  Detail & Related papers  (2026-02-18T20:25:13Z)
• Extension of Symmetrized Neural Network Operators with Fractional and Mixed Activation Functions [0.0]
  We propose a novel extension to symmetrized neural network operators by incorporating fractional and mixed activation functions.<n>Our framework introduces a fractional exponent in the activation functions, allowing adaptive non-linear approximations with improved accuracy.
  arXiv  Detail & Related papers  (2025-01-17T14:24:25Z)
• Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations [0.0]
  Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs)<n>These limitations impact their accuracy for problems involving rapid oscillations, sharp gradients, and complex boundary behaviors.<n>We investigate learnable activation functions as a solution to these challenges.
  arXiv  Detail & Related papers  (2024-11-22T18:25:13Z)
• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• Approximation of RKHS Functionals by Neural Networks [30.42446856477086]
  We study the approximation of functionals on kernel reproducing Hilbert spaces (RKHS's) using neural networks. We derive explicit error bounds for those induced by inverse multiquadric, Gaussian, and Sobolev kernels. We apply our findings to functional regression, proving that neural networks can accurately approximate the regression maps.
  arXiv  Detail & Related papers  (2024-03-18T18:58:23Z)
• Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
  The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
  arXiv  Detail & Related papers  (2023-04-17T14:23:43Z)
• Data-aware customization of activation functions reduces neural network error [0.35172332086962865]
  We show that data-aware customization of activation functions can result in striking reductions in neural network error. A simple substitution with the seagull'' activation function in an already-refined neural network can lead to an order-of-magnitude reduction in error.
  arXiv  Detail & Related papers  (2023-01-16T23:38:37Z)
• On the Activation Function Dependence of the Spectral Bias of Neural Networks [0.0]
  We study the phenomenon from the point of view of the spectral bias of neural networks. We provide a theoretical explanation for the spectral bias of ReLU neural networks by leveraging connections with the theory of finite element methods. We show that neural networks with the Hat activation function are trained significantly faster using gradient descent and ADAM.
  arXiv  Detail & Related papers  (2022-08-09T17:40:57Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Can Temporal-Difference and Q-Learning Learn Representation? A Mean-Field Theory [110.99247009159726]
  Temporal-difference and Q-learning play a key role in deep reinforcement learning, where they are empowered by expressive nonlinear function approximators such as neural networks. In particular, temporal-difference learning converges when the function approximator is linear in a feature representation, which is fixed throughout learning, and possibly diverges otherwise.
  arXiv  Detail & Related papers  (2020-06-08T17:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.