TempoRL: Learning When to Act

• URL: http://arxiv.org/abs/2106.05262v1
• Date: Wed, 9 Jun 2021 17:58:31 GMT
• Title: TempoRL: Learning When to Act
• Abstract summary: Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment. We propose a proactive setting in which the agent selects an action in a state and also for how long to commit to that action. Our TempoRL approach introduces skip connections between states and learns a skip-policy for repeating the same action along these skips. We demonstrate the effectiveness of TempoRL on a variety of traditional and deep RL environments, showing that our approach is capable of learning successful policies up to an order of magnitude faster than vanilla Q-learning.
• Score: 34.8990562818147
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning is a powerful approach to learn behaviour through interactions with an environment. However, behaviours are usually learned in a purely reactive fashion, where an appropriate action is selected based on an observation. In this form, it is challenging to learn when it is necessary to execute new decisions. This makes learning inefficient, especially in environments that need various degrees of fine and coarse control. To address this, we propose a proactive setting in which the agent not only selects an action in a state but also for how long to commit to that action. Our TempoRL approach introduces skip connections between states and learns a skip-policy for repeating the same action along these skips. We demonstrate the effectiveness of TempoRL on a variety of traditional and deep RL environments, showing that our approach is capable of learning successful policies up to an order of magnitude faster than vanilla Q-learning.

Related papers

• When to Plan: Learning to Select Between Reactive Control and Deliberative Planning [11.723839544878244]
  We introduce an RL method for training a meta-reasoning policy that allocates computation by conditioning on a reactive-policy uncertainty score.<n>We conduct an empirical study on motion planning and navigation environments, showing that this design enables the meta-reasoning policy to learn when the reactive policy provides a good-enough action versus when decision-time planning is needed.
  arXiv  Detail & Related papers  (2026-07-17T18:13:38Z)
• PRISE: LLM-Style Sequence Compression for Learning Temporal Action Abstractions in Control [55.81022882408587]
  Temporal action abstractions, along with belief state representations, are a powerful knowledge sharing mechanism for sequential decision making. We propose a novel view that treats inducing temporal action abstractions as a sequence compression problem. We introduce an approach that combines continuous action quantization with byte pair encoding to learn powerful action abstractions.
  arXiv  Detail & Related papers  (2024-02-16T04:55:09Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Inapplicable Actions Learning for Knowledge Transfer in Reinforcement Learning [3.194414753332705]
  We show that learning inapplicable actions greatly improves the sample efficiency of RL algorithms. Thanks to the transferability of the knowledge acquired, it can be reused in other tasks and domains to make the learning process more efficient.
  arXiv  Detail & Related papers  (2022-11-28T17:45:39Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)
• Learning Routines for Effective Off-Policy Reinforcement Learning [0.0]
  We propose a novel framework for reinforcement learning that effectively lifts such constraints. Within our framework, agents learn effective behavior over a routine space. We show that the resulting agents obtain relevant performance improvements while requiring fewer interactions with the environment per episode.
  arXiv  Detail & Related papers  (2021-06-05T18:41:57Z)
• NEARL: Non-Explicit Action Reinforcement Learning for Robotic Control [15.720231070808696]
  In this paper, we propose a novel hierarchical reinforcement learning framework without explicit action. Our meta policy tries to manipulate the next optimal state and actual action is produced by the inverse dynamics model. Under our framework, widely available state-only demonstrations can be exploited effectively for imitation learning.
  arXiv  Detail & Related papers  (2020-11-02T15:28:19Z)
• Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning [0.0]
  We show how a pre-trained task-independent language model can make a goal-conditional RL agent more sample efficient. We do this by facilitating transfer learning between different related tasks.
  arXiv  Detail & Related papers  (2020-07-10T06:42:00Z)
• Reinforcement Learning with Iterative Reasoning for Merging in Dense Traffic [41.46201285202203]
  Maneuvering in dense traffic is a challenging task for autonomous vehicles. We propose a combination of reinforcement learning and game theory to learn merging behaviors.
  arXiv  Detail & Related papers  (2020-05-25T02:57:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.