Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural
Networks to Recover Nonlinear Oscillations Efficiently
- URL: http://arxiv.org/abs/2304.09759v1
- Date: Tue, 18 Apr 2023 14:08:15 GMT
- Title: Amplifying Sine Unit: An Oscillatory Activation Function for Deep Neural
Networks to Recover Nonlinear Oscillations Efficiently
- Authors: Jamshaid Ul Rahman, Faiza Makhdoom, Dianchen Lu
- Abstract summary: In this work, we put forward a methodology based on deep neural networks with responsive layers structure to deal nonlinear oscillations in microelectromechanical systems.
We have proposed a novel oscillatory activation function called Amplifying Sine Unit denoted as ASU which is more efficient than GCU for complex vibratory systems.
Results show that the designed network with our proposed activation function ASU is more reliable and robust to handle the challenges posed by nonlinearity and oscillations.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Many industrial and real life problems exhibit highly nonlinear periodic
behaviors and the conventional methods may fall short of finding their
analytical or closed form solutions. Such problems demand some cutting edge
computational tools with increased functionality and reduced cost. Recently,
deep neural networks have gained massive research interest due to their ability
to handle large data and universality to learn complex functions. In this work,
we put forward a methodology based on deep neural networks with responsive
layers structure to deal nonlinear oscillations in microelectromechanical
systems. We incorporated some oscillatory and non oscillatory activation
functions such as growing cosine unit known as GCU, Sine, Mish and Tanh in our
designed network to have a comprehensive analysis on their performance for
highly nonlinear and vibrational problems. Integrating oscillatory activation
functions with deep neural networks definitely outperform in predicting the
periodic patterns of underlying systems. To support oscillatory actuation for
nonlinear systems, we have proposed a novel oscillatory activation function
called Amplifying Sine Unit denoted as ASU which is more efficient than GCU for
complex vibratory systems such as microelectromechanical systems. Experimental
results show that the designed network with our proposed activation function
ASU is more reliable and robust to handle the challenges posed by nonlinearity
and oscillations. To validate the proposed methodology, outputs of our networks
are being compared with the results from Livermore solver for ordinary
differential equation called LSODA. Further, graphical illustrations of
incurred errors are also being presented in the work.
Related papers
- KAN/MultKAN with Physics-Informed Spline fitting (KAN-PISF) for ordinary/partial differential equation discovery of nonlinear dynamic systems [0.0]
There is a dire need to interpret the machine learning models to develop a physical understanding of dynamic systems.
In this study, an equation discovery framework is proposed that includes i) sequentially regularized derivatives for denoising (SRDD) algorithm to denoise the measure data, ii) KAN to identify the equation structure and suggest relevant nonlinear functions.
arXiv Detail & Related papers (2024-11-18T18:14:51Z) - Nonlinear Neural Dynamics and Classification Accuracy in Reservoir Computing [3.196204482566275]
We study the accuracy of a reservoir computer in artificial classification tasks of varying complexity.
We find that, even for activation functions with extremely reduced nonlinearity, weak recurrent interactions and small input signals, the reservoir is able to compute useful representations.
arXiv Detail & Related papers (2024-11-15T08:52:12Z) - How neural networks learn to classify chaotic time series [77.34726150561087]
We study the inner workings of neural networks trained to classify regular-versus-chaotic time series.
We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
arXiv Detail & Related papers (2023-06-04T08:53:27Z) - Globally Optimal Training of Neural Networks with Threshold Activation
Functions [63.03759813952481]
We study weight decay regularized training problems of deep neural networks with threshold activations.
We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
arXiv Detail & Related papers (2023-03-06T18:59:13Z) - On Robust Numerical Solver for ODE via Self-Attention Mechanism [82.95493796476767]
We explore training efficient and robust AI-enhanced numerical solvers with a small data size by mitigating intrinsic noise disturbances.
We first analyze the ability of the self-attention mechanism to regulate noise in supervised learning and then propose a simple-yet-effective numerical solver, Attr, which introduces an additive self-attention mechanism to the numerical solution of differential equations.
arXiv Detail & Related papers (2023-02-05T01:39:21Z) - Signal Detection in MIMO Systems with Hardware Imperfections: Message
Passing on Neural Networks [101.59367762974371]
In this paper, we investigate signal detection in multiple-input-multiple-output (MIMO) communication systems with hardware impairments.
It is difficult to train a deep neural network (DNN) with limited pilot signals, hindering its practical applications.
We design an efficient message passing based Bayesian signal detector, leveraging the unitary approximate message passing (UAMP) algorithm.
arXiv Detail & Related papers (2022-10-08T04:32:58Z) - Momentum Diminishes the Effect of Spectral Bias in Physics-Informed
Neural Networks [72.09574528342732]
Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs)
They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias.
In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
arXiv Detail & Related papers (2022-06-29T19:03:10Z) - Temporal support vectors for spiking neuronal networks [0.0]
We introduce a novel extension of the static Support Vector Machine (T-SVM)
We show that T-SVM and its kernel extensions generate robust synaptic weight vectors in spiking neurons.
We propose T-SVM with nonlinear kernels as a new model of the computational role of the nonlinearities and extensive morphologies of neuronal dendritic trees.
arXiv Detail & Related papers (2022-05-28T23:47:15Z) - Neural Galerkin Schemes with Active Learning for High-Dimensional
Evolution Equations [44.89798007370551]
This work proposes Neural Galerkin schemes based on deep learning that generate training data with active learning for numerically solving high-dimensional partial differential equations.
Neural Galerkin schemes build on the Dirac-Frenkel variational principle to train networks by minimizing the residual sequentially over time.
Our finding is that the active form of gathering training data of the proposed Neural Galerkin schemes is key for numerically realizing the expressive power of networks in high dimensions.
arXiv Detail & Related papers (2022-03-02T19:09:52Z) - Machine Learning Link Inference of Noisy Delay-coupled Networks with
Opto-Electronic Experimental Tests [1.0766846340954257]
We devise a machine learning technique to solve the general problem of inferring network links that have time-delays.
We first train a type of machine learning system known as reservoir computing to mimic the dynamics of the unknown network.
We formulate and test a technique that uses the trained parameters of the reservoir system output layer to deduce an estimate of the unknown network structure.
arXiv Detail & Related papers (2020-10-29T00:24:13Z) - Lipschitz Recurrent Neural Networks [100.72827570987992]
We show that our Lipschitz recurrent unit is more robust with respect to input and parameter perturbations as compared to other continuous-time RNNs.
Our experiments demonstrate that the Lipschitz RNN can outperform existing recurrent units on a range of benchmark tasks.
arXiv Detail & Related papers (2020-06-22T08:44:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.