Learning and Understanding a Disentangled Feature Representation for
Hidden Parameters in Reinforcement Learning
- URL: http://arxiv.org/abs/2211.16315v1
- Date: Tue, 29 Nov 2022 15:55:30 GMT
- Title: Learning and Understanding a Disentangled Feature Representation for
Hidden Parameters in Reinforcement Learning
- Authors: Christopher Reale and Rebecca Russell
- Abstract summary: We present an unsupervised method to map RL trajectories into a feature space where distance represents the relative difference in system behavior due to hidden parameters.
Our approach disentangles the effects of hidden parameters by leveraging a recurrent neural network (RNN) world model as used in model-based RL.
- Score: 1.3909388235627789
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Hidden parameters are latent variables in reinforcement learning (RL)
environments that are constant over the course of a trajectory. Understanding
what, if any, hidden parameters affect a particular environment can aid both
the development and appropriate usage of RL systems. We present an unsupervised
method to map RL trajectories into a feature space where distance represents
the relative difference in system behavior due to hidden parameters. Our
approach disentangles the effects of hidden parameters by leveraging a
recurrent neural network (RNN) world model as used in model-based RL. First, we
alter the standard world model training algorithm to isolate the hidden
parameter information in the world model memory. Then, we use a metric learning
approach to map the RNN memory into a space with a distance metric
approximating a bisimulation metric with respect to the hidden parameters. The
resulting disentangled feature space can be used to meaningfully relate
trajectories to each other and analyze the hidden parameter. We demonstrate our
approach on four hidden parameters across three RL environments. Finally we
present two methods to help identify and understand the effects of hidden
parameters on systems.
Related papers
- Winning Prize Comes from Losing Tickets: Improve Invariant Learning by
Exploring Variant Parameters for Out-of-Distribution Generalization [76.27711056914168]
Out-of-Distribution (OOD) Generalization aims to learn robust models that generalize well to various environments without fitting to distribution-specific features.
Recent studies based on Lottery Ticket Hypothesis (LTH) address this problem by minimizing the learning target to find some of the parameters that are critical to the task.
We propose Exploring Variant parameters for Invariant Learning (EVIL) which also leverages the distribution knowledge to find the parameters that are sensitive to distribution shift.
arXiv Detail & Related papers (2023-10-25T06:10:57Z) - Seeking Neural Nuggets: Knowledge Transfer in Large Language Models from a Parametric Perspective [106.92016199403042]
We empirically investigate knowledge transfer from larger to smaller models through a parametric perspective.
We employ sensitivity-based techniques to extract and align knowledge-specific parameters between different large language models.
Our findings highlight the critical factors contributing to the process of parametric knowledge transfer.
arXiv Detail & Related papers (2023-10-17T17:58:34Z) - Deep Learning of Dynamical System Parameters from Return Maps as Images [0.0]
We present a novel approach to system identification using deep learning techniques.
We use a supervised learning approach for estimating the parameters of discrete and continuous-time dynamical systems.
arXiv Detail & Related papers (2023-06-20T03:23:32Z) - Efficient Parametric Approximations of Neural Network Function Space
Distance [6.117371161379209]
It is often useful to compactly summarize important properties of model parameters and training data so that they can be used later without storing and/or iterating over the entire dataset.
We consider estimating the Function Space Distance (FSD) over a training set, i.e. the average discrepancy between the outputs of two neural networks.
We propose a Linearized Activation TRick (LAFTR) and derive an efficient approximation to FSD for ReLU neural networks.
arXiv Detail & Related papers (2023-02-07T15:09:23Z) - Symmetry Detection in Trajectory Data for More Meaningful Reinforcement
Learning Representations [0.0]
We present a method of automatically detecting RL symmetries directly from raw trajectory data without requiring active control of the system.
We show in experiments on two simulated RL use cases that our method can determine the symmetries underlying both the environment physics and the trained RL policy.
arXiv Detail & Related papers (2022-11-29T17:00:26Z) - Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
We construct a dataset of neural network checkpoints and train a generative model on the parameters.
We find that our approach successfully generates parameters for a wide range of loss prompts.
We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
arXiv Detail & Related papers (2022-09-26T17:59:58Z) - Efficient Continual Adaptation for Generative Adversarial Networks [97.20244383723853]
We present a continual learning approach for generative adversarial networks (GANs)
Our approach is based on learning a set of global and task-specific parameters.
We show that the feature-map transformation based approach outperforms state-of-the-art continual GANs methods.
arXiv Detail & Related papers (2021-03-06T05:09:37Z) - Learning Invariant Representations for Reinforcement Learning without
Reconstruction [98.33235415273562]
We study how representation learning can accelerate reinforcement learning from rich observations, such as images, without relying either on domain knowledge or pixel-reconstruction.
Bisimulation metrics quantify behavioral similarity between states in continuous MDPs.
We demonstrate the effectiveness of our method at disregarding task-irrelevant information using modified visual MuJoCo tasks.
arXiv Detail & Related papers (2020-06-18T17:59:35Z) - On Parameter Tuning in Meta-learning for Computer Vision [2.3513645401551333]
In this paper, we investigate mage recognition for unseen categories of a given dataset with limited training information.
We deploy a zero-shot learning (ZSL) algorithm to achieve this goal.
We also explore the effect of parameter tuning on performance of semantic auto-encoder (SAE)
arXiv Detail & Related papers (2020-02-11T15:07:30Z) - Meta Reinforcement Learning with Autonomous Inference of Subtask
Dependencies [57.27944046925876]
We propose and address a novel few-shot RL problem, where a task is characterized by a subtask graph.
Instead of directly learning a meta-policy, we develop a Meta-learner with Subtask Graph Inference.
Our experiment results on two grid-world domains and StarCraft II environments show that the proposed method is able to accurately infer the latent task parameter.
arXiv Detail & Related papers (2020-01-01T17:34:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.