Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPs

• URL: http://arxiv.org/abs/2411.07832v1
• Date: Tue, 12 Nov 2024 14:27:45 GMT
• Title: Dynamical-VAE-based Hindsight to Learn the Causal Dynamics of Factored-POMDPs
• Authors: Chao Han, Debabrota Basu, Michael Mangan, Eleni Vasilaki, Aditya Gilra,
• Abstract summary: We introduce a Dynamical Variational Auto-Encoder (DVAE) designed to learn causal Markovian dynamics from offline trajectories. Our method employs an extended hindsight framework that integrates past, current, and multi-step future information. Empirical results reveal that this approach uncovers the causal graph governing hidden state transitions more effectively than history-based and typical hindsight-based models.
• Score: 9.662551514840388
• License:
• Abstract: Learning representations of underlying environmental dynamics from partial observations is a critical challenge in machine learning. In the context of Partially Observable Markov Decision Processes (POMDPs), state representations are often inferred from the history of past observations and actions. We demonstrate that incorporating future information is essential to accurately capture causal dynamics and enhance state representations. To address this, we introduce a Dynamical Variational Auto-Encoder (DVAE) designed to learn causal Markovian dynamics from offline trajectories in a POMDP. Our method employs an extended hindsight framework that integrates past, current, and multi-step future information within a factored-POMDP setting. Empirical results reveal that this approach uncovers the causal graph governing hidden state transitions more effectively than history-based and typical hindsight-based models.

Related papers

• Remember and Recall: Associative-Memory-based Trajectory Prediction [25.349986959111757]
  We propose the Fragmented-Memory-based Trajectory Prediction (FMTP) model, inspired by the remarkable learning capabilities of humans. The FMTP model employs discrete representations to enhance computational efficiency by reducing information redundancy. We develop an advanced reasoning engine based on language models to deeply learn the associative rules among these discrete representations.
  arXiv  Detail & Related papers  (2024-10-03T04:32:21Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Intrinsic Dynamics-Driven Generalizable Scene Representations for Vision-Oriented Decision-Making Applications [0.21051221444478305]
  How to improve the ability of scene representation is a key issue in vision-oriented decision-making applications. We propose an intrinsic dynamics-driven representation learning method with sequence models in visual reinforcement learning.
  arXiv  Detail & Related papers  (2024-05-30T06:31:03Z)
• Modeling Spatio-temporal Dynamical Systems with Neural Discrete Learning and Levels-of-Experts [33.335735613579914]
  We address the issue of modeling and estimating changes in the state oftemporal- dynamical systems based on a sequence of observations like video frames. This paper propose the universal expert module -- that is, optical flow estimation component, to capture the laws of general physical processes in a data-driven fashion. We conduct extensive experiments and ablations to demonstrate that the proposed framework achieves large performance margins, compared with the existing SOTA baselines.
  arXiv  Detail & Related papers  (2024-02-06T06:27:07Z)
• Exploring Model Transferability through the Lens of Potential Energy [78.60851825944212]
  Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models. Existing methods for measuring the transferability of pre-trained models rely on statistical correlations between encoded static features and task labels. We present an insightful physics-inspired approach named PED to address these challenges.
  arXiv  Detail & Related papers  (2023-08-29T07:15:57Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL [90.06845886194235]
  We propose a modified objective for model-based reinforcement learning (RL) We integrate a term inspired by variational empowerment into a state-space model based on mutual information. We evaluate the approach on a suite of vision-based robot control tasks with natural video backgrounds.
  arXiv  Detail & Related papers  (2022-04-18T23:09:23Z)
• Accelerating Representation Learning with View-Consistent Dynamics in Data-Efficient Reinforcement Learning [12.485293708638292]
  We propose to accelerate state representation learning by enforcing view-consistency on the dynamics. We introduce a formalism of Multi-view Markov Decision Process (MMDP) that incorporates multiple views of the state. Following the structure of MMDP, our method, View-Consistent Dynamics (VCD), learns state representations by training a view-consistent dynamics model in the latent space.
  arXiv  Detail & Related papers  (2022-01-18T14:28:30Z)
• Meta-learning using privileged information for dynamics [66.32254395574994]
  We extend the Neural ODE Process model to use additional information within the Learning Using Privileged Information setting. We validate our extension with experiments showing improved accuracy and calibration on simulated dynamics tasks.
  arXiv  Detail & Related papers  (2021-04-29T12:18:02Z)
• Episodic Memory for Learning Subjective-Timescale Models [1.933681537640272]
  In model-based learning, an agent's model is commonly defined over transitions between consecutive states of an environment. In contrast, intelligent behaviour in biological organisms is characterised by the ability to plan over varying temporal scales depending on the context. We devise a novel approach to learning a transition dynamics model, based on the sequences of episodic memories that define the agent's subjective timescale.
  arXiv  Detail & Related papers  (2020-10-03T21:55:40Z)
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning [124.9856253431878]
  We decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.
  arXiv  Detail & Related papers  (2020-05-14T08:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.