Learning Task-Driven Control Policies via Information Bottlenecks
- URL: http://arxiv.org/abs/2002.01428v1
- Date: Tue, 4 Feb 2020 17:50:06 GMT
- Title: Learning Task-Driven Control Policies via Information Bottlenecks
- Authors: Vincent Pacelli and Anirudha Majumdar
- Abstract summary: This paper presents a reinforcement learning approach to synthesizing task-driven control policies for robotic systems equipped with rich sensory modalities.
Standard reinforcement learning algorithms typically produce policies that tightly couple control actions to the entirety of the system's state and rich sensor observations.
In contrast, the approach we present here learns to create a task-driven representation that is used to compute control actions.
- Score: 7.271970309320002
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper presents a reinforcement learning approach to synthesizing
task-driven control policies for robotic systems equipped with rich sensory
modalities (e.g., vision or depth). Standard reinforcement learning algorithms
typically produce policies that tightly couple control actions to the entirety
of the system's state and rich sensor observations. As a consequence, the
resulting policies can often be sensitive to changes in task-irrelevant
portions of the state or observations (e.g., changing background colors). In
contrast, the approach we present here learns to create a task-driven
representation that is used to compute control actions. Formally, this is
achieved by deriving a policy gradient-style algorithm that creates an
information bottleneck between the states and the task-driven representation;
this constrains actions to only depend on task-relevant information. We
demonstrate our approach in a thorough set of simulation results on multiple
examples including a grasping task that utilizes depth images and a
ball-catching task that utilizes RGB images. Comparisons with a standard policy
gradient approach demonstrate that the task-driven policies produced by our
algorithm are often significantly more robust to sensor noise and
task-irrelevant changes in the environment.
Related papers
- Imagination Policy: Using Generative Point Cloud Models for Learning Manipulation Policies [25.760946763103483]
We propose Imagination Policy, a novel multi-task key-frame policy network for solving high-precision pick and place tasks.
Instead of learning actions directly, Imagination Policy generates point clouds to imagine desired states which are then translated to actions using rigid action estimation.
arXiv Detail & Related papers (2024-06-17T17:00:41Z) - Learning Generalizable Manipulation Policies with Object-Centric 3D
Representations [65.55352131167213]
GROOT is an imitation learning method for learning robust policies with object-centric and 3D priors.
It builds policies that generalize beyond their initial training conditions for vision-based manipulation.
GROOT's performance excels in generalization over background changes, camera viewpoint shifts, and the presence of new object instances.
arXiv Detail & Related papers (2023-10-22T18:51:45Z) - Learning Robust Policy against Disturbance in Transition Dynamics via
State-Conservative Policy Optimization [63.75188254377202]
Deep reinforcement learning algorithms can perform poorly in real-world tasks due to discrepancy between source and target environments.
We propose a novel model-free actor-critic algorithm to learn robust policies without modeling the disturbance in advance.
Experiments in several robot control tasks demonstrate that SCPO learns robust policies against the disturbance in transition dynamics.
arXiv Detail & Related papers (2021-12-20T13:13:05Z) - Distilling Motion Planner Augmented Policies into Visual Control
Policies for Robot Manipulation [26.47544415550067]
We propose to distill a state-based motion planner augmented policy to a visual control policy.
We evaluate our method on three manipulation tasks in obstructed environments.
Our framework is highly sample-efficient and outperforms the state-of-the-art algorithms.
arXiv Detail & Related papers (2021-11-11T18:52:00Z) - Trajectory-based Reinforcement Learning of Non-prehensile Manipulation
Skills for Semi-Autonomous Teleoperation [18.782289957834475]
We present a semi-autonomous teleoperation framework for a pick-and-place task using an RGB-D sensor.
A trajectory-based reinforcement learning is utilized for learning the non-prehensile manipulation to rearrange the objects.
We show that the proposed method outperforms manual keyboard control in terms of the time duration for the grasping.
arXiv Detail & Related papers (2021-09-27T14:27:28Z) - Composable Learning with Sparse Kernel Representations [110.19179439773578]
We present a reinforcement learning algorithm for learning sparse non-parametric controllers in a Reproducing Kernel Hilbert Space.
We improve the sample complexity of this approach by imposing a structure of the state-action function through a normalized advantage function.
We demonstrate the performance of this algorithm on learning obstacle-avoidance policies in multiple simulations of a robot equipped with a laser scanner while navigating in a 2D environment.
arXiv Detail & Related papers (2021-03-26T13:58:23Z) - Policy Information Capacity: Information-Theoretic Measure for Task
Complexity in Deep Reinforcement Learning [83.66080019570461]
We propose two environment-agnostic, algorithm-agnostic quantitative metrics for task difficulty.
We show that these metrics have higher correlations with normalized task solvability scores than a variety of alternatives.
These metrics can also be used for fast and compute-efficient optimizations of key design parameters.
arXiv Detail & Related papers (2021-03-23T17:49:50Z) - Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients [54.98496284653234]
We consider the task of training a policy that maximizes reward while minimizing disclosure of certain sensitive state variables through the actions.
We solve this problem by introducing a regularizer based on the mutual information between the sensitive state and the actions.
We develop a model-based estimator for optimization of privacy-constrained policies.
arXiv Detail & Related papers (2020-12-30T03:22:35Z) - Learning Adaptive Exploration Strategies in Dynamic Environments Through
Informed Policy Regularization [100.72335252255989]
We study the problem of learning exploration-exploitation strategies that effectively adapt to dynamic environments.
We propose a novel algorithm that regularizes the training of an RNN-based policy using informed policies trained to maximize the reward in each task.
arXiv Detail & Related papers (2020-05-06T16:14:48Z) - Robotic Arm Control and Task Training through Deep Reinforcement
Learning [6.249276977046449]
We show that Trust Region Policy Optimization and DeepQ-Network with Normalized Advantage Functions perform better than Deep Deterministic Policy Gradient and Vanilla Policy Gradient.
Real-world experiments let show that our polices, if correctly trained on simulation, can be transferred and executed in a real environment with almost no changes.
arXiv Detail & Related papers (2020-05-06T07:34:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.