Generalising Discrete Action Spaces with Conditional Action Trees

• URL: http://arxiv.org/abs/2104.07294v1
• Date: Thu, 15 Apr 2021 08:10:18 GMT
• Title: Generalising Discrete Action Spaces with Conditional Action Trees
• Authors: Christopher Bamford, Alvaro Ovalle
• Abstract summary: We introduce em Conditional Action Trees with two main objectives. We show several proof-of-concept experiments ranging from environments with discrete action spaces to those with large action spaces commonly found in RTS-style games.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: There are relatively few conventions followed in reinforcement learning (RL) environments to structure the action spaces. As a consequence the application of RL algorithms to tasks with large action spaces with multiple components require additional effort to adjust to different formats. In this paper we introduce {\em Conditional Action Trees} with two main objectives: (1) as a method of structuring action spaces in RL to generalise across several action space specifications, and (2) to formalise a process to significantly reduce the action space by decomposing it into multiple sub-spaces, favoring a multi-staged decision making approach. We show several proof-of-concept experiments validating our scheme, ranging from environments with basic discrete action spaces to those with large combinatorial action spaces commonly found in RTS-style games.

Related papers

• Offline Reinforcement Learning With Combinatorial Action Spaces [12.904199719046968]
  Reinforcement learning problems often involve large action spaces arising from the simultaneous execution of multiple sub-actions. We propose Branch Value Estimation (BVE), which effectively captures sub-action dependencies and scales to large spaces by learning to evaluate only a small subset of actions at each timestep. Our experiments show that BVE outperforms state-of-the-art methods across a range of action space sizes.
  arXiv  Detail & Related papers  (2024-10-28T15:49:46Z)
• Solving Continual Offline RL through Selective Weights Activation on Aligned Spaces [52.649077293256795]
  Continual offline reinforcement learning (CORL) has shown impressive ability in diffusion-based lifelong learning systems. We propose Vector-Quantized Continual diffuser, named VQ-CD, to break the barrier of different spaces between various tasks.
  arXiv  Detail & Related papers  (2024-10-21T07:13:45Z)
• In-Context Reinforcement Learning for Variable Action Spaces [46.29510499540938]
  Headless-AD is capable of generalizing to discrete action spaces of variable size, semantic content and order. We show that Headless-AD exhibits significant capability to generalize to action spaces it has never encountered.
  arXiv  Detail & Related papers  (2023-12-20T16:58:55Z)
• Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion [86.6191592951269]
  Merging models fine-tuned from common extensively pretrained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multitask model that performs well across diverse tasks. We propose the CONtinuous relaxation dis (Concrete) subspace learning method to identify a common lowdimensional subspace and utilize its shared information track interference problem without sacrificing performance.
  arXiv  Detail & Related papers  (2023-12-11T07:24:54Z)
• AI planning in the imagination: High-level planning on learned abstract search spaces [68.75684174531962]
  We propose a new method, called PiZero, that gives an agent the ability to plan in an abstract search space that the agent learns during training. We evaluate our method on multiple domains, including the traveling salesman problem, Sokoban, 2048, the facility location problem, and Pacman.
  arXiv  Detail & Related papers  (2023-08-16T22:47:16Z)
• Dynamic Neighborhood Construction for Structured Large Discrete Action Spaces [2.285821277711785]
  Large discrete action spaces (LDAS) remain a central challenge in reinforcement learning. Existing solution approaches can handle unstructured LDAS with up to a few million actions. We propose Dynamic Neighborhood Construction (DNC), a novel exploitation paradigm for SLDAS.
  arXiv  Detail & Related papers  (2023-05-31T14:26:14Z)
• Adaptive Discretization using Voronoi Trees for Continuous-Action POMDPs [7.713622698801596]
  We propose a new sampling-based online POMDP solver, called Adaptive Discretization using Voronoi Trees (ADVT) ADVT uses Monte Carlo Tree Search in combination with an adaptive discretization of the action space as well as optimistic optimization. Experiments on simulations of four types of benchmark problems indicate that ADVT outperforms and scales substantially better to high-dimensional continuous action spaces.
  arXiv  Detail & Related papers  (2022-09-13T05:04:49Z)
• Graph Convolutional Module for Temporal Action Localization in Videos [142.5947904572949]
  We claim that the relations between action units play an important role in action localization. A more powerful action detector should not only capture the local content of each action unit but also allow a wider field of view on the context related to it. We propose a general graph convolutional module (GCM) that can be easily plugged into existing action localization methods.
  arXiv  Detail & Related papers  (2021-12-01T06:36:59Z)
• Reinforcement Learning in Factored Action Spaces using Tensor Decompositions [92.05556163518999]
  We propose a novel solution for Reinforcement Learning (RL) in large, factored action spaces using tensor decompositions. We use cooperative multi-agent reinforcement learning scenario as the exemplary setting.
  arXiv  Detail & Related papers  (2021-10-27T15:49:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.