Central and Non-central Limit Theorems arising from the Scattering Transform and its Neural Activation Generalization

• URL: http://arxiv.org/abs/2011.10801v1
• Date: Sat, 21 Nov 2020 14:31:57 GMT
• Title: Central and Non-central Limit Theorems arising from the Scattering Transform and its Neural Activation Generalization
• Abstract summary: We study a generalization of the scattering transform (ST) that includes broad neural activation functions, which is called neural activation ST (NAST) Here, the nonlinear function models how a neuron gets excited by the input signal. Results explain how NAST processes complicated and non-stationary time series, and pave a way towards statistical inference based on NAST under the non-null case.
• Score: 4.125187280299247
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Motivated by analyzing complicated and non-stationary time series, we study a generalization of the scattering transform (ST) that includes broad neural activation functions, which is called neural activation ST (NAST). On the whole, NAST is a transform that comprises a sequence of ``neural processing units'', each of which applies a high pass filter to the input from the previous layer followed by a composition with a nonlinear function as the output to the next neuron. Here, the nonlinear function models how a neuron gets excited by the input signal. In addition to showing properties like non-expansion, horizontal translational invariability and insensitivity to local deformation, the statistical properties of the second order NAST of a Gaussian process with various dependence and (non-)stationarity structure and its interaction with the chosen high pass filters and activation functions are explored and central limit theorem (CLT) and non-CLT results are provided. Numerical simulations are also provided. The results explain how NAST processes complicated and non-stationary time series, and pave a way towards statistical inference based on NAST under the non-null case.

Related papers

• Revisiting Neural Processes via Fourier Transform and Volterra Series [8.061821280531884]
  Neural processes (NPs) are a family of probabilistic functional models.<n>NPs face two limitations: (i) they stack generic components with non-linearities, obscuring the induced function class and limiting interpretability.<n>We propose two conditional NPs (CNPs): SFConvCNPs, which stack SFConv blocks with non-linearities, and SFVConvCNPs, which integrate the Volterra formulation.
  arXiv  Detail & Related papers  (2026-05-31T11:27:48Z)
• Neural-POD: A Plug-and-Play Neural Operator Framework for Infinite-Dimensional Functional Nonlinear Proper Orthogonal Decomposition [7.1950116347185995]
  We propose the Neural Proper Orthogonal Decomposition (Neural-POD), a plug-and-play neural operator framework.<n>Neural-POD formulates basis construction as a sequence of residual minimization problems solved through neural network training.<n>We demonstrate the robustness of Neural-POD with different complex resolutions, including the Burgers' and NavierStokes equations.
  arXiv  Detail & Related papers  (2026-02-17T15:01:40Z)
• From Taylor Series to Fourier Synthesis: The Periodic Linear Unit [0.0]
  We introduce the Periodic Linear Unit (PLU), a learnable sine-wave based activation with periodic non-monotonicity.<n>We demonstrate that a minimal activation with only two neurons can solve the spiral classification task, a feat impossible for equivalent networks using standard activations.
  arXiv  Detail & Related papers  (2025-08-02T03:26:48Z)
• Analysis of Fourier Neural Operators via Effective Field Theory [1.7102697561186413]
  We present the first systematic effective-field-theory analysis of FNOs in an infinite-dimensional function space.<n>We show that nonlinear activations inevitably couple frequency inputs to high-frequency modes that are otherwise discarded by spectral truncation.
  arXiv  Detail & Related papers  (2025-07-29T14:10:46Z)
• Provable In-Context Learning of Nonlinear Regression with Transformers [58.018629320233174]
  In-context learning (ICL) is the ability to perform unseen tasks using task-specific prompts without updating parameters.<n>Recent research has actively explored the training dynamics behind ICL.<n>This paper investigates more complex nonlinear regression tasks, aiming to uncover how transformers acquire in-context learning capabilities.
  arXiv  Detail & Related papers  (2025-07-28T00:09:28Z)
• Generative System Dynamics in Recurrent Neural Networks [56.958984970518564]
  We investigate the continuous time dynamics of Recurrent Neural Networks (RNNs) We show that skew-symmetric weight matrices are fundamental to enable stable limit cycles in both linear and nonlinear configurations. Numerical simulations showcase how nonlinear activation functions not only maintain limit cycles, but also enhance the numerical stability of the system integration process.
  arXiv  Detail & Related papers  (2025-04-16T10:39:43Z)
• Spectral Convolutional Conditional Neural Processes [4.52069311861025]
  Conditional Neural Processes (CNPs) constitute a family of probabilistic models that harness the flexibility of neural networks to parameterize processes. We propose Spectral Convolutional Conditional Neural Processes (SConvCNPs), a new addition to the NPs family that allows for more efficient representation of functions in the frequency domain.
  arXiv  Detail & Related papers  (2024-04-19T21:13:18Z)
• Variational Sampling of Temporal Trajectories [39.22854981703244]
  We introduce a mechanism to learn the distribution of trajectories by parameterizing the transition function $f$ explicitly as an element in a function space. Our framework allows efficient synthesis of novel trajectories, while also directly providing a convenient tool for inference.
  arXiv  Detail & Related papers  (2024-03-18T02:12:12Z)
• Non Commutative Convolutional Signal Models in Neural Networks: Stability to Small Deformations [111.27636893711055]
  We study the filtering and stability properties of non commutative convolutional filters. Our results have direct implications for group neural networks, multigraph neural networks and quaternion neural networks.
  arXiv  Detail & Related papers  (2023-10-05T20:27:22Z)
• MP-GELU Bayesian Neural Networks: Moment Propagation by GELU Nonlinearity [0.0]
  We propose a novel nonlinear function named moment propagating-Gaussian error linear unit (MP-GELU) that enables the fast derivation of first and second moments in BNNs. MP-GELU provides higher prediction accuracy and better quality of uncertainty with faster execution than those of ReLU-based BNNs.
  arXiv  Detail & Related papers  (2022-11-24T03:37:29Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Uncertainty Propagation in Convolutional Neural Networks: Technical Report [13.224071661974593]
  We study the problem of propagation of uncertainty through typical building blocks of a Convolutional Neural Network (CNN) These include layers that perform linear operations, as well as layers that act non-linearly on their input, such as the Rectified Linear Unit (ReLU)
  arXiv  Detail & Related papers  (2021-02-11T15:17:46Z)
• Probabilistic Numeric Convolutional Neural Networks [80.42120128330411]
  Continuous input signals like images and time series that are irregularly sampled or have missing values are challenging for existing deep learning methods. We propose Probabilistic Convolutional Neural Networks which represent features as Gaussian processes (GPs) We then define a convolutional layer as the evolution of a PDE defined on this GP, followed by a nonlinearity. In experiments we show that our approach yields a $3times$ reduction of error from the previous state of the art on the SuperPixel-MNIST dataset and competitive performance on the medical time2012 dataset PhysioNet.
  arXiv  Detail & Related papers  (2020-10-21T10:08:21Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Measuring Model Complexity of Neural Networks with Curve Activation Functions [100.98319505253797]
  We propose the linear approximation neural network (LANN) to approximate a given deep model with curve activation function. We experimentally explore the training process of neural networks and detect overfitting. We find that the $L1$ and $L2$ regularizations suppress the increase of model complexity.
  arXiv  Detail & Related papers  (2020-06-16T07:38:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.