Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight Campaign Design

• URL: http://arxiv.org/abs/2103.08981v1
• Date: Tue, 16 Mar 2021 11:17:02 GMT
• Title: Hierarchical Reinforcement Learning Framework for Stochastic Spaceflight Campaign Design
• Authors: Yuji Takubo, Hao Chen, and Koki Ho
• Abstract summary: This paper develops a hierarchical reinforcement learning architecture for spaceflight campaign design under uncertainty. It is applied to a set of human lunar exploration campaign scenarios with uncertain in-situ resource utilization (ISRU) performance.
• Score: 5.381116150823982
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper develops a hierarchical reinforcement learning architecture for multi-mission spaceflight campaign design under uncertainty, including vehicle design, infrastructure deployment planning, and space transportation scheduling. This problem involves a high-dimensional design space and is challenging especially with uncertainty present. To tackle this challenge, the developed framework has a hierarchical structure with reinforcement learning (RL) and network-based mixed-integer linear programming (MILP), where the former optimizes campaign-level decisions (e.g., design of the vehicle used throughout the campaign, destination demand assigned to each mission in the campaign), whereas the latter optimizes the detailed mission-level decisions (e.g., when to launch what from where to where). The framework is applied to a set of human lunar exploration campaign scenarios with uncertain in-situ resource utilization (ISRU) performance as a case study. The main value of this work is its integration of the rapidly growing RL research and the existing MILP-based space logistics methods through a hierarchical framework to handle the otherwise intractable complexity of space mission design under uncertainty. We expect this unique framework to be a critical steppingstone for the emerging research direction of artificial intelligence for space mission design.

Related papers

• Self-reconfiguration Strategies for Space-distributed Spacecraft [17.70060501010008]
  This paper proposes a distributed on-orbit spacecraft assembly algorithm, where future spacecraft can assemble modules with different functions on orbit. Reasonable and efficient on-orbit self-reconfiguration algorithms play a crucial role in realizing the benefits of distributed spacecraft.
  arXiv  Detail & Related papers  (2024-11-26T06:05:44Z)
• LLMSat: A Large Language Model-Based Goal-Oriented Agent for Autonomous Space Exploration [0.0]
  This work explores the application of Large Language Models (LLMs) as the high-level control system of a spacecraft. A series of deep space mission scenarios simulated within the popular game engine Kerbal Space Program are used as case studies to evaluate the implementation against the requirements.
  arXiv  Detail & Related papers  (2024-04-13T03:33:17Z)
• Long-HOT: A Modular Hierarchical Approach for Long-Horizon Object Transport [83.06265788137443]
  We address key challenges in long-horizon embodied exploration and navigation by proposing a new object transport task and a novel modular framework for temporally extended navigation. Our first contribution is the design of a novel Long-HOT environment focused on deep exploration and long-horizon planning. We propose a modular hierarchical transport policy (HTP) that builds a topological graph of the scene to perform exploration with the help of weighted frontiers.
  arXiv  Detail & Related papers  (2022-10-28T05:30:49Z)
• Overcoming Exploration: Deep Reinforcement Learning in Complex Environments from Temporal Logic Specifications [2.8904578737516764]
  We present a Deep Reinforcement Learning (DRL) algorithm for a task-guided robot with unknown continuous-time dynamics deployed in a large-scale complex environment. Our framework is shown to significantly improve performance (effectiveness, efficiency) and exploration of robots tasked with complex missions in large-scale complex environments.
  arXiv  Detail & Related papers  (2022-01-28T16:39:08Z)
• Successor Feature Landmarks for Long-Horizon Goal-Conditioned Reinforcement Learning [54.378444600773875]
  We introduce Successor Feature Landmarks (SFL), a framework for exploring large, high-dimensional environments. SFL drives exploration by estimating state-novelty and enables high-level planning by abstracting the state-space as a non-parametric landmark-based graph. We show in our experiments on MiniGrid and ViZDoom that SFL enables efficient exploration of large, high-dimensional state spaces.
  arXiv  Detail & Related papers  (2021-11-18T18:36:05Z)
• Design Strategy Network: A deep hierarchical framework to represent generative design strategies in complex action spaces [0.0]
  This work introduces Design Strategy Network (DSN), a data-driven deep hierarchical framework that learns strategies over arbitrary complex action spaces. The hierarchical architecture decomposes every action decision into first predicting a preferred spatial region in the design space. Results show that DSNs significantly outperform non-hierarchical methods of policy representation.
  arXiv  Detail & Related papers  (2021-10-07T19:29:40Z)
• Landmark Policy Optimization for Object Navigation Task [77.34726150561087]
  This work studies object goal navigation task, which involves navigating to the closest object related to the given semantic category in unseen environments. Recent works have shown significant achievements both in the end-to-end Reinforcement Learning approach and modular systems, but need a big step forward to be robust and optimal. We propose a hierarchical method that incorporates standard task formulation and additional area knowledge as landmarks, with a way to extract these landmarks.
  arXiv  Detail & Related papers  (2021-09-17T12:28:46Z)
• Temporal Predictive Coding For Model-Based Planning In Latent Space [80.99554006174093]
  We present an information-theoretic approach that employs temporal predictive coding to encode elements in the environment that can be predicted across time. We evaluate our model on a challenging modification of standard DMControl tasks where the background is replaced with natural videos that contain complex but irrelevant information to the planning task.
  arXiv  Detail & Related papers  (2021-06-14T04:31:15Z)
• Constrained optimisation of preliminary spacecraft configurations under the design-for-demise paradigm [1.0205541448656992]
  Most mid-sized satellites currently launched and already in orbit fail to comply with the casualty risk threshold of 0.0001. Satellites manufacturers and mission operators need to perform a disposal through a controlled re-entry. This additional cost and complexity can be removed as the spacecraft is directly compliant with the casualty risk regulations.
  arXiv  Detail & Related papers  (2020-12-27T17:48:29Z)
• Reinforcement Learning for Low-Thrust Trajectory Design of Interplanetary Missions [77.34726150561087]
  This paper investigates the use of reinforcement learning for the robust design of interplanetary trajectories in presence of severe disturbances. An open-source implementation of the state-of-the-art algorithm Proximal Policy Optimization is adopted. The resulting Guidance and Control Network provides both a robust nominal trajectory and the associated closed-loop guidance law.
  arXiv  Detail & Related papers  (2020-08-19T15:22:15Z)
• Jump Operator Planning: Goal-Conditioned Policy Ensembles and Zero-Shot Transfer [71.44215606325005]
  We propose a novel framework called Jump-Operator Dynamic Programming for quickly computing solutions within a super-exponential space of sequential sub-goal tasks. This approach involves controlling over an ensemble of reusable goal-conditioned polices functioning as temporally extended actions. We then identify classes of objective functions on this subspace whose solutions are invariant to the grounding, resulting in optimal zero-shot transfer.
  arXiv  Detail & Related papers  (2020-07-06T05:13:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.