Rational Neural Networks have Expressivity Advantages

• URL: http://arxiv.org/abs/2602.12390v1
• Date: Thu, 12 Feb 2026 20:33:42 GMT
• Title: Rational Neural Networks have Expressivity Advantages
• Abstract summary: We study neural networks with trainable low-degree rational activation functions.<n>We show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations.
• Score: 6.180619136976129
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study neural networks with trainable low-degree rational activation functions and show that they are more expressive and parameter-efficient than modern piecewise-linear and smooth activations such as ELU, LeakyReLU, LogSigmoid, PReLU, ReLU, SELU, CELU, Sigmoid, SiLU, Mish, Softplus, Tanh, Softmin, Softmax, and LogSoftmax. For an error target of $\varepsilon>0$, we establish approximation-theoretic separations: Any network built from standard fixed activations can be uniformly approximated on compact domains by a rational-activation network with only $\mathrm{poly}(\log\log(1/\varepsilon))$ overhead in size, while the converse provably requires $Ω(\log(1/\varepsilon))$ parameters in the worst case. This exponential gap persists at the level of full networks and extends to gated activations and transformer-style nonlinearities. In practice, rational activations integrate seamlessly into standard architectures and training pipelines, allowing rationals to match or outperform fixed activations under identical architectures and optimizers.

Related papers

• Rethinking Neural Nonlinearity as Gating [0.7073210405344709]
  We show that standard activations are in fact instances of a single Threshold Gating primitive.<n>For softmax, we show that it admits an exact TG conversion via its equivalent per-element Sigmoid form.<n>We also propose a 'Minimal Branch Theorem' which relates the minimum number of required branches in our primitive to the trainability of general deep neural networks.
  arXiv  Detail & Related papers  (2026-07-03T09:38:16Z)
• Smoothness Adaptivity in Constant-Depth Neural Networks: Optimal Rates via Smooth Activations [11.1057367593811]
  We study approximation and statistical properties of neural networks with smooth activations for learning functions in the Sobolev space.<n>We prove that constant-depth networks equipped with smooth activations achieve smoothness adaptivity.<n>We identify activation smoothness as a fundamental mechanism, complementary to depth, for achieving optimal rates over Sobolev function classes.
  arXiv  Detail & Related papers  (2026-02-23T10:38:12Z)
• Efficient High-Accuracy PDEs Solver with the Linear Attention Neural Operator [2.1595890347557756]
  We present a novel type of neural operators, Linear Attention Neural Operator (LANO)<n>LANO achieves both scalability and high accuracy by reformulating attention through an agent-based mechanism.<n> Empirically, LANO surpasses current state-of-the-art neural PDE solvers, including Transolver with slice-based softmax attention.
  arXiv  Detail & Related papers  (2025-10-19T13:03:09Z)
• Topology-Aware Activation Functions in Neural Networks [0.0]
  This study explores novel activation functions that enhance the ability of neural networks to manipulate data topology during training.<n>We propose $mathrmSmoothSplit$ and $mathrmParametricSplit$, which introduce topology "cutting" capabilities.<n>Our findings highlight the potential of topology-aware activation functions in advancing neural network architectures.
  arXiv  Detail & Related papers  (2025-07-17T07:48:36Z)
• Extension of Symmetrized Neural Network Operators with Fractional and Mixed Activation Functions [0.0]
  We propose a novel extension to symmetrized neural network operators by incorporating fractional and mixed activation functions.<n>Our framework introduces a fractional exponent in the activation functions, allowing adaptive non-linear approximations with improved accuracy.
  arXiv  Detail & Related papers  (2025-01-17T14:24:25Z)
• HyperZ$\cdot$Z$\cdot$W Operator Connects Slow-Fast Networks for Full Context Interaction [0.0]
  Self-attention mechanism utilizes large implicit weight matrices, programmed through dot product-based activations with very few trainable parameters, to enable long sequence modeling. In this paper, we investigate the possibility of discarding residual learning by employing large implicit kernels to achieve full context interaction at each layer of the network. Our model incorporates several innovative components and exhibits excellent properties, such as introducing local feedback error for updating the slow network, stable zero-mean features, faster training convergence, and fewer model parameters.
  arXiv  Detail & Related papers  (2024-01-31T15:57:21Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Moderate Adaptive Linear Units (MoLU) [0.0]
  We propose a novel activation function for deep neural networks, defined analytically as: f(x)=x times (1+tanh(x))/2.<n>MoLU combines mathematical elegance with empirical effectiveness, exhibiting superior performance in terms of prediction accuracy, convergence speed, and computational efficiency.
  arXiv  Detail & Related papers  (2023-02-27T11:55:24Z)
• Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach [64.23331120621118]
  This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks. We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network. We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
  arXiv  Detail & Related papers  (2022-08-08T03:13:24Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)
• Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds [99.23098204458336]
  Certified robustness is a desirable property for deep neural networks in safety-critical applications. We show that our method consistently outperforms state-of-the-art methods on MNIST and TinyNet datasets.
  arXiv  Detail & Related papers  (2021-11-02T06:44:10Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.