Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
- URL: http://arxiv.org/abs/2010.01381v1
- Date: Sat, 3 Oct 2020 16:15:22 GMT
- Title: Cubic Spline Smoothing Compensation for Irregularly Sampled Sequences
- Authors: Jing Shi, Jing Bi, Yingru Liu, Chenliang Xu
- Abstract summary: The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences.
While ODE produces the smooth hidden states between observation intervals, the RNN will trigger a hidden state jump when a new observation arrives.
We propose the cubic spline smoothing compensation, which is a stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end.
- Score: 42.27009455538217
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The marriage of recurrent neural networks and neural ordinary differential
networks (ODE-RNN) is effective in modeling irregularly-observed sequences.
While ODE produces the smooth hidden states between observation intervals, the
RNN will trigger a hidden state jump when a new observation arrives, thus cause
the interpolation discontinuity problem. To address this issue, we propose the
cubic spline smoothing compensation, which is a stand-alone module upon either
the output or the hidden state of ODE-RNN and can be trained end-to-end. We
derive its analytical solution and provide its theoretical interpolation error
bound. Extensive experiments indicate its merits over both ODE-RNN and cubic
spline interpolation.
Related papers
- Kernel Limit of Recurrent Neural Networks Trained on Ergodic Data Sequences [0.0]
We characterize the tangents of recurrent neural networks (RNNs) as the number of hidden units, data samples in the sequence, hidden state updates, and training steps simultaneously grow to infinity.
These methods give rise to the neural kernel (NTK) limits for RNNs trained on data sequences as the number of data samples and size of the neural network grow to infinity.
arXiv Detail & Related papers (2023-08-28T13:17:39Z) - From NeurODEs to AutoencODEs: a mean-field control framework for
width-varying Neural Networks [68.8204255655161]
We propose a new type of continuous-time control system, called AutoencODE, based on a controlled field that drives dynamics.
We show that many architectures can be recovered in regions where the loss function is locally convex.
arXiv Detail & Related papers (2023-07-05T13:26:17Z) - Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification.
Our results indicate that interpolating with smoother functions leads to better generalization.
arXiv Detail & Related papers (2023-05-30T19:37:44Z) - Learning Discretized Neural Networks under Ricci Flow [51.36292559262042]
We study Discretized Neural Networks (DNNs) composed of low-precision weights and activations.
DNNs suffer from either infinite or zero gradients due to the non-differentiable discrete function during training.
arXiv Detail & Related papers (2023-02-07T10:51:53Z) - A Neural Network-enhanced Reproducing Kernel Particle Method for
Modeling Strain Localization [0.0]
In this work, neural network-enhanced reproducing kernel particle method (NN-RKPM) is proposed.
The location, orientation, and shape of the solution transition near a localization is automatically captured by the NN approximation.
The effectiveness of the proposed NN-RKPM is verified by a series of numerical verifications.
arXiv Detail & Related papers (2022-04-28T23:59:38Z) - Framing RNN as a kernel method: A neural ODE approach [11.374487003189468]
We show that the solution of a RNN can be viewed as a linear function of a specific feature set of the input sequence, known as the signature.
We obtain theoretical guarantees on generalization and stability for a large class of recurrent networks.
arXiv Detail & Related papers (2021-06-02T14:46:40Z) - Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and
(gradient) stable architecture for learning long time dependencies [15.2292571922932]
We propose a novel architecture for recurrent neural networks.
Our proposed RNN is based on a time-discretization of a system of second-order ordinary differential equations.
Experiments show that the proposed RNN is comparable in performance to the state of the art on a variety of benchmarks.
arXiv Detail & Related papers (2020-10-02T12:35:04Z) - Multipole Graph Neural Operator for Parametric Partial Differential
Equations [57.90284928158383]
One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data.
We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity.
Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
arXiv Detail & Related papers (2020-06-16T21:56:22Z) - Measuring Model Complexity of Neural Networks with Curve Activation
Functions [100.98319505253797]
We propose the linear approximation neural network (LANN) to approximate a given deep model with curve activation function.
We experimentally explore the training process of neural networks and detect overfitting.
We find that the $L1$ and $L2$ regularizations suppress the increase of model complexity.
arXiv Detail & Related papers (2020-06-16T07:38:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.