Learning World Models with Identifiable Factorization
- URL: http://arxiv.org/abs/2306.06561v2
- Date: Tue, 27 Jun 2023 12:58:29 GMT
- Title: Learning World Models with Identifiable Factorization
- Authors: Yu-Ren Liu, Biwei Huang, Zhengmao Zhu, Honglong Tian, Mingming Gong,
Yang Yu, Kun Zhang
- Abstract summary: We propose IFactor to model four distinct categories of latent state variables.
Our analysis establishes block-wise identifiability of these latent variables.
We present a practical approach to learning the world model with identifiable blocks.
- Score: 39.767120163665574
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Extracting a stable and compact representation of the environment is crucial
for efficient reinforcement learning in high-dimensional, noisy, and
non-stationary environments. Different categories of information coexist in
such environments -- how to effectively extract and disentangle these
information remains a challenging problem. In this paper, we propose IFactor, a
general framework to model four distinct categories of latent state variables
that capture various aspects of information within the RL system, based on
their interactions with actions and rewards. Our analysis establishes
block-wise identifiability of these latent variables, which not only provides a
stable and compact representation but also discloses that all reward-relevant
factors are significant for policy learning. We further present a practical
approach to learning the world model with identifiable blocks, ensuring the
removal of redundants but retaining minimal and sufficient information for
policy optimization. Experiments in synthetic worlds demonstrate that our
method accurately identifies the ground-truth latent variables, substantiating
our theoretical findings. Moreover, experiments in variants of the DeepMind
Control Suite and RoboDesk showcase the superior performance of our approach
over baselines.
Related papers
- Where Do We Stand with Implicit Neural Representations? A Technical and Performance Survey [16.89460694470542]
Implicit Neural Representations (INRs) have emerged as a paradigm in knowledge representation.
INRs leverage multilayer perceptrons (MLPs) to model data as continuous implicit functions.
This survey introduces a clear taxonomy that categorises them into four key areas: activation functions, position encoding, combined strategies, and network structure.
arXiv Detail & Related papers (2024-11-06T06:14:24Z) - Exploring the Precise Dynamics of Single-Layer GAN Models: Leveraging Multi-Feature Discriminators for High-Dimensional Subspace Learning [0.0]
We study the training dynamics of a single-layer GAN model from the perspective of subspace learning.
By bridging our analysis to the realm of subspace learning, we systematically compare the efficacy of GAN-based methods against conventional approaches.
arXiv Detail & Related papers (2024-11-01T10:21:12Z) - Evaluating the Stability of Deep Learning Latent Feature Spaces [0.0]
This study introduces a novel workflow to evaluate the stability of latent spaces, ensuring consistency and reliability in subsequent analyses.
We implement this workflow across 500 autoencoder realizations and three datasets, encompassing both synthetic and real-world scenarios.
Our findings highlight inherent instabilities in latent feature spaces and demonstrate the workflow's efficacy in quantifying and interpreting these instabilities.
arXiv Detail & Related papers (2024-02-17T23:41:15Z) - Epistemic Exploration for Generalizable Planning and Learning in Non-Stationary Settings [23.038187032666304]
This paper introduces a new approach for continual planning and model learning in non-stationary environments.
The proposed framework models gaps in the agent's current state of knowledge and uses them to conduct focused, investigative explorations.
Empirical evaluations on several non-stationary benchmark domains show that this approach significantly outperforms planning and RL baselines in terms of sample complexity.
arXiv Detail & Related papers (2024-02-13T00:50:06Z) - Leveraging sparse and shared feature activations for disentangled
representation learning [112.22699167017471]
We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation.
We validate our approach on six real world distribution shift benchmarks, and different data modalities.
arXiv Detail & Related papers (2023-04-17T01:33:24Z) - Latent Variable Representation for Reinforcement Learning [131.03944557979725]
It remains unclear theoretically and empirically how latent variable models may facilitate learning, planning, and exploration to improve the sample efficiency of model-based reinforcement learning.
We provide a representation view of the latent variable models for state-action value functions, which allows both tractable variational learning algorithm and effective implementation of the optimism/pessimism principle.
In particular, we propose a computationally efficient planning algorithm with UCB exploration by incorporating kernel embeddings of latent variable models.
arXiv Detail & Related papers (2022-12-17T00:26:31Z) - Variational Distillation for Multi-View Learning [104.17551354374821]
We design several variational information bottlenecks to exploit two key characteristics for multi-view representation learning.
Under rigorously theoretical guarantee, our approach enables IB to grasp the intrinsic correlation between observations and semantic labels.
arXiv Detail & Related papers (2022-06-20T03:09:46Z) - Information-Theoretic Odometry Learning [83.36195426897768]
We propose a unified information theoretic framework for learning-motivated methods aimed at odometry estimation.
The proposed framework provides an elegant tool for performance evaluation and understanding in information-theoretic language.
arXiv Detail & Related papers (2022-03-11T02:37:35Z) - Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
We propose a novel approach to identify meaningful and independent factors of variation in a dataset.
Our method involves two separate latent subspaces for the target property and the remaining input information.
We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
arXiv Detail & Related papers (2021-11-25T17:33:12Z) - Action-Sufficient State Representation Learning for Control with
Structural Constraints [21.47086290736692]
In this paper, we focus on partially observable environments and propose to learn a minimal set of state representations that capture sufficient information for decision-making.
We build a generative environment model for the structural relationships among variables in the system and present a principled way to characterize ASRs.
Our empirical results on CarRacing and VizDoom demonstrate a clear advantage of learning and using ASRs for policy learning.
arXiv Detail & Related papers (2021-10-12T03:16:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.