Investigating the Impact of Observation Space Design Choices On Training Reinforcement Learning Solutions for Spacecraft Problems

• URL: http://arxiv.org/abs/2501.06016v1
• Date: Fri, 10 Jan 2025 14:53:21 GMT
• Title: Investigating the Impact of Observation Space Design Choices On Training Reinforcement Learning Solutions for Spacecraft Problems
• Authors: Nathaniel Hamilton, Kyle Dunlap, Kerianne L Hobbs,
• Abstract summary: This paper focuses on how changes to the environment's observation space can impact the training and performance of RL agents learning the spacecraft inspection task. The first looks at the impact of sensors that were designed to help agents learn the task. The second looks at the impact of reference frames, reorienting the agent to see the world from a different perspective.
• Score: 0.3441021278275805
• License:
• Abstract: Recent research using Reinforcement Learning (RL) to learn autonomous control for spacecraft operations has shown great success. However, a recent study showed their performance could be improved by changing the action space, i.e. control outputs, used in the learning environment. This has opened the door for finding more improvements through further changes to the environment. The work in this paper focuses on how changes to the environment's observation space can impact the training and performance of RL agents learning the spacecraft inspection task. The studies are split into two groups. The first looks at the impact of sensors that were designed to help agents learn the task. The second looks at the impact of reference frames, reorienting the agent to see the world from a different perspective. The results show the sensors are not necessary, but most of them help agents learn more optimal behavior, and that the reference frame does not have a large impact, but is best kept consistent.

Related papers

• Variable-Agnostic Causal Exploration for Reinforcement Learning [56.52768265734155]
  We introduce a novel framework, Variable-Agnostic Causal Exploration for Reinforcement Learning (VACERL) Our approach automatically identifies crucial observation-action steps associated with key variables using attention mechanisms. It constructs the causal graph connecting these steps, which guides the agent towards observation-action pairs with greater causal influence on task completion.
  arXiv  Detail & Related papers  (2024-07-17T09:45:27Z)
• Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning [58.69297999175239]
  In robot learning, the observation space is crucial due to the distinct characteristics of different modalities. In this study, we explore the influence of various observation spaces on robot learning, focusing on three predominant modalities: RGB, RGB-D, and point cloud.
  arXiv  Detail & Related papers  (2024-02-04T14:18:45Z)
• Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery [64.41455104593304]
  Reinforcement Learning from Human Feedback (RLHF) can improve machine learning models and align them with human preferences. We propose to adapt similar RL-based methods to unsupervised object discovery. We demonstrate that our approach is not only more accurate, but also orders of magnitudes faster to train.
  arXiv  Detail & Related papers  (2023-10-29T17:03:12Z)
• Information Design in Multi-Agent Reinforcement Learning [61.140924904755266]
  Reinforcement learning (RL) is inspired by the way human infants and animals learn from the environment. Research in computational economics distills two ways to influence others directly: by providing tangible goods (mechanism design) and by providing information (information design)
  arXiv  Detail & Related papers  (2023-05-08T07:52:15Z)
• Basis for Intentions: Efficient Inverse Reinforcement Learning using Past Experience [89.30876995059168]
  inverse reinforcement learning (IRL) -- inferring the reward function of an agent from observing its behavior. This paper addresses the problem of IRL -- inferring the reward function of an agent from observing its behavior.
  arXiv  Detail & Related papers  (2022-08-09T17:29:49Z)
• Causal Influence Detection for Improving Efficiency in Reinforcement Learning [11.371889042789219]
  We introduce a measure of situation-dependent causal influence based on conditional mutual information. We show that it can reliably detect states of influence. All modified algorithms show strong increases in data efficiency on robotic manipulation tasks.
  arXiv  Detail & Related papers  (2021-06-07T09:21:56Z)
• LASER: Learning a Latent Action Space for Efficient Reinforcement Learning [41.53297694894669]
  We present LASER, a method to learn latent action spaces for efficient reinforcement learning. We show improved sample efficiency compared to the original action space from better alignment of the action space to the task space, as we observe with visualizations of the learned action space manifold.
  arXiv  Detail & Related papers  (2021-03-29T17:40:02Z)
• Causal Curiosity: RL Agents Discovering Self-supervised Experiments for Causal Representation Learning [24.163616087447874]
  We introduce em causal curiosity, a novel intrinsic reward. We show that it allows our agents to learn optimal sequences of actions. We also show that the knowledge of causal factor representations aids zero-shot learning for more complex tasks.
  arXiv  Detail & Related papers  (2020-10-07T02:07:51Z)
• Action Space Shaping in Deep Reinforcement Learning [7.508516104014916]
  Reinforcement learning has been successful in training agents in various learning environments, including video-games. We aim to gain insight on these action space modifications by conducting extensive experiments in video-game environments. Our results show how domain-specific removal of actions and discretization of continuous actions can be crucial for successful learning.
  arXiv  Detail & Related papers  (2020-04-02T13:25:55Z)
• Mutual Information-based State-Control for Intrinsically Motivated Reinforcement Learning [102.05692309417047]
  In reinforcement learning, an agent learns to reach a set of goals by means of an external reward signal. In the natural world, intelligent organisms learn from internal drives, bypassing the need for external signals. We propose to formulate an intrinsic objective as the mutual information between the goal states and the controllable states.
  arXiv  Detail & Related papers  (2020-02-05T19:21:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.