Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability

• URL: http://arxiv.org/abs/2409.16824v1
• Date: Wed, 25 Sep 2024 11:22:29 GMT
• Title: Uncertainty Representations in State-Space Layers for Deep Reinforcement Learning under Partial Observability
• Authors: Carlos E. Luis, Alessandro G. Bottero, Julia Vinogradska, Felix Berkenkamp, Jan Peters,
• Abstract summary: We propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan. Experiments show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.
• Score: 59.758009422067
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Optimal decision-making under partial observability requires reasoning about the uncertainty of the environment's hidden state. However, most reinforcement learning architectures handle partial observability with sequence models that have no internal mechanism to incorporate uncertainty in their hidden state representation, such as recurrent neural networks, deterministic state-space models and transformers. Inspired by advances in probabilistic world models for reinforcement learning, we propose a standalone Kalman filter layer that performs closed-form Gaussian inference in linear state-space models and train it end-to-end within a model-free architecture to maximize returns. Similar to efficient linear recurrent layers, the Kalman filter layer processes sequential data using a parallel scan, which scales logarithmically with the sequence length. By design, Kalman filter layers are a drop-in replacement for other recurrent layers in standard model-free architectures, but importantly they include an explicit mechanism for probabilistic filtering of the latent state representation. Experiments in a wide variety of tasks with partial observability show that Kalman filter layers excel in problems where uncertainty reasoning is key for decision-making, outperforming other stateful models.

Related papers

• Outlier-robust Kalman Filtering through Generalised Bayes [45.51425214486509]
  We derive a novel, provably robust, and closed-form Bayesian update rule for online filtering in state-space models. Our method matches or outperforms other robust filtering methods at a much lower computational cost.
  arXiv  Detail & Related papers  (2024-05-09T09:40:56Z)
• Last layer state space model for representation learning and uncertainty quantification [0.0]
  We propose to decompose a classification or regression task in two steps: a representation learning stage to learn low-dimensional states, and a state space model for uncertainty estimation. We demonstrate how predictive distributions can be estimated on top of an existing and trained neural network, by adding a state space-based last layer. Our model accounts for the noisy data structure, due to unknown or unavailable variables, and is able to provide confidence intervals on predictions.
  arXiv  Detail & Related papers  (2023-07-04T08:37:37Z)
• Kalman Filter for Online Classification of Non-Stationary Data [101.26838049872651]
  In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. We introduce a probabilistic Bayesian online learning model by using a neural representation and a state space model over the linear predictor weights. In experiments in multi-class classification we demonstrate the predictive ability of the model and its flexibility to capture non-stationarity.
  arXiv  Detail & Related papers  (2023-06-14T11:41:42Z)
• On Uncertainty in Deep State Space Models for Model-Based Reinforcement Learning [21.63642325390798]
  We show that RSSMs use a suboptimal inference scheme and that models trained using this inference overestimate the aleatoric uncertainty of the ground truth system. We propose an alternative approach building on well-understood components for modeling aleatoric and epistemic uncertainty, dubbed Variational Recurrent Kalman Network (VRKN) Our experiments show that using the VRKN instead of the RSSM improves performance in tasks where appropriately capturing aleatoric uncertainty is crucial.
  arXiv  Detail & Related papers  (2022-10-17T16:59:48Z)
• Robust and Provably Monotonic Networks [0.0]
  We present a new method to constrain the Lipschitz constant of dense deep learning models. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor decays in the LHCb realtime data-processing system.
  arXiv  Detail & Related papers  (2021-11-30T19:01:32Z)
• Unsupervised Learned Kalman Filtering [84.18625250574853]
  unsupervised adaptation is achieved by exploiting the hybrid model-based/data-driven architecture of KalmanNet. We numerically demonstrate that when the noise statistics are unknown, unsupervised KalmanNet achieves a similar performance to KalmanNet with supervised learning.
  arXiv  Detail & Related papers  (2021-10-18T04:04:09Z)
• KalmanNet: Neural Network Aided Kalman Filtering for Partially Known Dynamics [84.18625250574853]
  We present KalmanNet, a real-time state estimator that learns from data to carry out Kalman filtering under non-linear dynamics. We numerically demonstrate that KalmanNet overcomes nonlinearities and model mismatch, outperforming classic filtering methods.
  arXiv  Detail & Related papers  (2021-07-21T12:26:46Z)
• Neural Kalman Filtering [62.997667081978825]
  We show that a gradient-descent approximation to the Kalman filter requires only local computations with variance weighted prediction errors. We also show that it is possible under the same scheme to adaptively learn the dynamics model with a learning rule that corresponds directly to Hebbian plasticity.
  arXiv  Detail & Related papers  (2021-02-19T16:43:15Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.