Deep Subspace Encoders for Nonlinear System Identification
- URL: http://arxiv.org/abs/2210.14816v2
- Date: Wed, 5 Jul 2023 13:52:23 GMT
- Title: Deep Subspace Encoders for Nonlinear System Identification
- Authors: Gerben I. Beintema, Maarten Schoukens, Roland T\'oth
- Abstract summary: We propose a method which uses a truncated prediction loss and a subspace encoder for state estimation.
We show that, under mild conditions, the proposed method is locally consistent, increases optimization stability, and achieves increased data efficiency.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Using Artificial Neural Networks (ANN) for nonlinear system identification
has proven to be a promising approach, but despite of all recent research
efforts, many practical and theoretical problems still remain open.
Specifically, noise handling and models, issues of consistency and reliable
estimation under minimisation of the prediction error are the most severe
problems. The latter comes with numerous practical challenges such as explosion
of the computational cost in terms of the number of data samples and the
occurrence of instabilities during optimization. In this paper, we aim to
overcome these issues by proposing a method which uses a truncated prediction
loss and a subspace encoder for state estimation. The truncated prediction loss
is computed by selecting multiple truncated subsections from the time series
and computing the average prediction loss. To obtain a computationally
efficient estimation method that minimizes the truncated prediction loss, a
subspace encoder represented by an artificial neural network is introduced.
This encoder aims to approximate the state reconstructability map of the
estimated model to provide an initial state for each truncated subsection given
past inputs and outputs. By theoretical analysis, we show that, under mild
conditions, the proposed method is locally consistent, increases optimization
stability, and achieves increased data efficiency by allowing for overlap
between the subsections. Lastly, we provide practical insights and user
guidelines employing a numerical example and state-of-the-art benchmark
results.
Related papers
- A sparse PAC-Bayesian approach for high-dimensional quantile prediction [0.0]
This paper presents a novel probabilistic machine learning approach for high-dimensional quantile prediction.
It uses a pseudo-Bayesian framework with a scaled Student-t prior and Langevin Monte Carlo for efficient computation.
Its effectiveness is validated through simulations and real-world data, where it performs competitively against established frequentist and Bayesian techniques.
arXiv Detail & Related papers (2024-09-03T08:01:01Z) - A Pseudo-Semantic Loss for Autoregressive Models with Logical
Constraints [87.08677547257733]
Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning.
We show how to maximize the likelihood of a symbolic constraint w.r.t the neural network's output distribution.
We also evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation.
arXiv Detail & Related papers (2023-12-06T20:58:07Z) - Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels [57.46832672991433]
We propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS)
We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises.
We develop an expectation-propagation expectation-maximization algorithm for efficient posterior inference and function estimation.
arXiv Detail & Related papers (2023-10-09T03:55:09Z) - Collapsed Inference for Bayesian Deep Learning [36.1725075097107]
We introduce a novel collapsed inference scheme that performs Bayesian model averaging using collapsed samples.
A collapsed sample represents uncountably many models drawn from the approximate posterior.
Our proposed use of collapsed samples achieves a balance between scalability and accuracy.
arXiv Detail & Related papers (2023-06-16T08:34:42Z) - Scalable computation of prediction intervals for neural networks via
matrix sketching [79.44177623781043]
Existing algorithms for uncertainty estimation require modifying the model architecture and training procedure.
This work proposes a new algorithm that can be applied to a given trained neural network and produces approximate prediction intervals.
arXiv Detail & Related papers (2022-05-06T13:18:31Z) - Gaining Outlier Resistance with Progressive Quantiles: Fast Algorithms
and Theoretical Studies [1.6457778420360534]
A framework of outlier-resistant estimation is introduced to robustify arbitrarily loss function.
A new technique is proposed to alleviate the requirement on starting point such that on regular datasets the number of data reestimations can be substantially reduced.
The obtained estimators, though not necessarily globally or even globally, enjoymax optimality in both low dimensions.
arXiv Detail & Related papers (2021-12-15T20:35:21Z) - Imputation-Free Learning from Incomplete Observations [73.15386629370111]
We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation.
We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation.
Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
arXiv Detail & Related papers (2021-07-05T12:44:39Z) - Nonlinear state-space identification using deep encoder networks [0.0]
This paper introduces a method that approximates the simulation loss by splitting the data set into independent sections to the multiple shooting method.
The main contribution is the introduction of an encoder function to estimate the initial state at the start each section.
arXiv Detail & Related papers (2020-12-14T16:49:06Z) - Amortized Conditional Normalized Maximum Likelihood: Reliable Out of
Distribution Uncertainty Estimation [99.92568326314667]
We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation.
Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle.
We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
arXiv Detail & Related papers (2020-11-05T08:04:34Z) - Learning Output Embeddings in Structured Prediction [73.99064151691597]
A powerful and flexible approach to structured prediction consists in embedding the structured objects to be predicted into a feature space of possibly infinite dimension.
A prediction in the original space is computed by solving a pre-image problem.
In this work, we propose to jointly learn a finite approximation of the output embedding and the regression function into the new feature space.
arXiv Detail & Related papers (2020-07-29T09:32:53Z) - Dimensionality reduction to maximize prediction generalization
capability [0.7614628596146599]
Generalization of time series prediction remains an important open issue in machine learning, wherein earlier methods have either large generalization error or local minima.
We develop an analytically solvable, unsupervised learning scheme that extracts the most informative components for predicting future inputs, termed predictive principal component analysis (PredPCA)
Our scheme can effectively remove unpredictable noise and minimize test prediction error through convex optimization.
arXiv Detail & Related papers (2020-03-01T12:04:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.