MPC-based Deep Reinforcement Learning Method for Space Robotic Control with Fuel Sloshing Mitigation

• URL: http://arxiv.org/abs/2509.21045v1
• Date: Thu, 25 Sep 2025 11:51:07 GMT
• Title: MPC-based Deep Reinforcement Learning Method for Space Robotic Control with Fuel Sloshing Mitigation
• Abstract summary: This paper presents an integrated Reinforcement Learning (RL) and Model Predictive Control (MPC) framework for autonomous satellite docking with a partially filled fuel tank.<n>To address this, we integrate PPO and SAC algorithms with MPC, leveraging MPC's predictive capabilities to accelerate RL training and improve control robustness.<n>This study advances fuel-efficient and disturbance-resilient satellite docking, enhancing the feasibility of on-orbit refueling and servicing missions.
• Score: 2.2423135771458194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents an integrated Reinforcement Learning (RL) and Model Predictive Control (MPC) framework for autonomous satellite docking with a partially filled fuel tank. Traditional docking control faces challenges due to fuel sloshing in microgravity, which induces unpredictable forces affecting stability. To address this, we integrate Proximal Policy Optimization (PPO) and Soft Actor-Critic (SAC) RL algorithms with MPC, leveraging MPC's predictive capabilities to accelerate RL training and improve control robustness. The proposed approach is validated through Zero-G Lab of SnT experiments for planar stabilization and high-fidelity numerical simulations for 6-DOF docking with fuel sloshing dynamics. Simulation results demonstrate that SAC-MPC achieves superior docking accuracy, higher success rates, and lower control effort, outperforming standalone RL and PPO-MPC methods. This study advances fuel-efficient and disturbance-resilient satellite docking, enhancing the feasibility of on-orbit refueling and servicing missions.

Related papers

• SRL-MPC: Shape-Aware Reinforcement Learned Model Predictive Control [64.66285242901918]
  Shape-Aware Reinforcement Learned Model Predictive Control (SRL-MPC) is a method for safe, efficient, and adaptive navigation in crowds with heterogeneous shapes without geometry simplification.<n>A reinforcement learning framework then learns a neural policy that reads geometric separation features (GSFs) and outputs real-time MPC parameter updates.<n> Experiments in randomized crowd scenarios with arbitrary shaped robot fleets demonstrate the effectiveness, scalability, and robustness of SRL-MPC.
  arXiv  Detail & Related papers  (2026-08-21T14:46:06Z)
• Joint Air Traffic Flow and Capacity Management via Answer Set Programming [64.47906587579985]
  We introduce a joint ATFCM model with an encoding in Answer Set Programming (ASP)<n>Our results indicate that the ASP model outperforms the MIP model.<n>While DAC has the largest improvement on solving performance compared to rerouting, unrestricted variants of DAC or rerouting lead to search space thrashing.
  arXiv  Detail & Related papers  (2026-06-22T07:58:54Z)
• Memory-Efficient Meta-Reinforcement Learning for Adaptive Safety-Critical Control in Adversarial Spacecraft Proximity Operations [42.47287159687486]
  This paper investigates the performance of three recurrent network architectures (LSTM, Gated Recurrent Unit (GRU), Selective State Space Model (Mamba)) and two training algorithms (Proximal Policy Optimization (PPO) and Soft Actor Critic (SAC)) to identify the best setup for tuning ICCBF class-K functions via meta-RL.<n>Results indicate that state space models such as Mamba when used with PPO achieve superior task completion, safety, and fuel-savings compared to other architectures.
  arXiv  Detail & Related papers  (2026-06-16T01:53:54Z)
• Prior Policy Guided Dual-Agent Coordinated Manipulation Planning of Spacecraft-Manipulator System [5.3085579284738085]
  We propose a Dual-Agent Coordinated Manipulation Planning framework.<n>It simultaneously achieves high-precision end-effector pose reaching for a 6-DoF space manipulator and attitude stabilization of the base spacecraft.
  arXiv  Detail & Related papers  (2026-05-25T02:32:23Z)
• Robust Adaptive Predictive Control for Hook-Based Aerial Transportation Between Moving Platforms [1.4592997284674396]
  This paper presents a novel model predictive control (MPC) approach for autonomous pick-and-place between moving platforms with a hook-equipped aerial manipulator.<n>The effectiveness of the proposed method is evaluated in complex simulated scenarios and in real-world flight experiments.
  arXiv  Detail & Related papers  (2026-05-04T09:14:34Z)
• Sim-to-reality adaptation for Deep Reinforcement Learning applied to an underwater docking application [0.0]
  Deep Reinforcement Learning (DRL) offers a robust alternative to traditional control methods for autonomous underwater docking.<n>This paper presents a systematic approach for autonomous docking using the Girona Autonomous Underwater Vehicle (AUV) by leveraging a high-fidelity digital twin environment.
  arXiv  Detail & Related papers  (2026-03-12T15:01:25Z)
• Stabilizing Policy Optimization via Logits Convexity [59.242732612484474]
  We show that the convexity of the supervised fine-tuning loss with respect to model logits plays a key role in enabling stable training.<n>Motivated by this observation, we propose Logits Convex Optimization (LCO), a simple yet effective policy optimization framework.
  arXiv  Detail & Related papers  (2026-03-01T07:40:12Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Optimal Multi-Debris Mission Planning in LEO: A Deep Reinforcement Learning Approach with Co-Elliptic Transfers and Refueling [22.261628532402067]
  This paper introduces a unified coelliptic maneuver framework that combines Hohmann transfers, safety proximity operations, and explicit refueling logic.<n>We benchmark three distinct planning algorithms Greedy, Monte Carlo Tree Search (MCTS), and deep reinforcement learning (RL)<n> Experimental results over 100 test scenarios demonstrate that Masked PPO achieves superior mission efficiency and computational performance.
  arXiv  Detail & Related papers  (2026-02-04T22:15:14Z)
• Shielded Controller Units for RL with Operational Constraints Applied to Remote Microgrids [50.64533198075622]
  Reinforcement learning (RL) is a powerful framework for optimizing decision-making in complex systems under uncertainty.<n>In this paper, we introduce Shielded Controller Units (SCUs), a systematic and interpretable approach that leverages prior knowledge of system dynamics.<n>We demonstrate the effectiveness of SCUs on a remote microgrid optimization task with strict operational requirements.
  arXiv  Detail & Related papers  (2025-11-30T19:28:34Z)
• ASTREA: Introducing Agentic Intelligence for Orbital Thermal Autonomy [51.56484100374058]
  ASTREA is the first agentic system deployed on flight-heritage hardware (TRL 9) for autonomous spacecraft operations.<n>We integrate a resource-constrained Large Language Model (LLM) agent with a reinforcement learning controller in an asynchronous architecture tailored for space-qualified platforms.
  arXiv  Detail & Related papers  (2025-09-16T08:52:13Z)
• Autonomous UAV Flight Navigation in Confined Spaces: A Reinforcement Learning Approach [0.686108371431346]
  Unmanned Aerial Vehicles (UAVs) offer a promising alternative, but GPS-denied environments require robust control policies to prevent collisions.<n>This paper provides a comparative study of two leading DRL algorithms for this task.<n>The training was conducted with procedurally generated duct environments in Genesis simulation environment.
  arXiv  Detail & Related papers  (2025-08-22T21:29:59Z)
• Intersection of Reinforcement Learning and Bayesian Optimization for Intelligent Control of Industrial Processes: A Safe MPC-based DPG using Multi-Objective BO [0.0]
  Model Predictive Control (MPC)-based Reinforcement Learning (RL) offers a structured and interpretable alternative to Deep Neural Network (DNN)-based RL methods.<n>Standard MPC-RL approaches often suffer from slow convergence, suboptimal policy learning due to limited parameterization, and safety issues during online adaptation.<n>We propose a novel framework that integrates MPC-RL with Multi-Objective Bayesian Optimization (MOBO)
  arXiv  Detail & Related papers  (2025-07-14T02:31:52Z)
• Neural-based Control for CubeSat Docking Maneuvers [0.0]
  This paper presents an innovative approach employing Artificial Neural Networks (ANN) trained through Reinforcement Learning (RL) The proposed strategy is easily implementable onboard and offers fast adaptability and robustness to disturbances by learning control policies from experience. Our findings highlight the efficacy of RL in assuring the adaptability and efficiency of spacecraft RVD, offering insights into future mission expectations.
  arXiv  Detail & Related papers  (2024-10-16T16:05:46Z)
• Rocket Landing Control with Random Annealing Jump Start Reinforcement Learning [19.87058922821708]
  This paper significantly elevates the success rate of rocket landing control from 8% with a baseline controller to 97% on a high-fidelity rocket model using reinforcement learning (RL) Our approach, called Random Annealing Jump Start (RAJS), is tailored for real-world goal-oriented problems by leveraging prior feedback controllers as guide policy to facilitate environmental exploration and policy learning in RL.
  arXiv  Detail & Related papers  (2024-07-21T07:47:53Z)
• Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Tasks [0.24578723416255746]
  In robotics, contemporary strategies are learning-based, characterized by a complex black-box nature and a lack of interpretability. We propose integrating a collision-free trajectory planner based on deep reinforcement learning (DRL) with a novel auto-tuning low-level control strategy.
  arXiv  Detail & Related papers  (2024-02-04T15:54:03Z)
• Confidence-Controlled Exploration: Efficient Sparse-Reward Policy Learning for Robot Navigation [72.24964965882783]
  Reinforcement learning (RL) is a promising approach for robotic navigation, allowing robots to learn through trial and error.<n>Real-world robotic tasks often suffer from sparse rewards, leading to inefficient exploration and suboptimal policies.<n>We introduce Confidence-Controlled Exploration (CCE), a novel method that improves sample efficiency in RL-based robotic navigation without modifying the reward function.
  arXiv  Detail & Related papers  (2023-06-09T18:45:15Z)
• Reinforcement Learning for Low-Thrust Trajectory Design of Interplanetary Missions [77.34726150561087]
  This paper investigates the use of reinforcement learning for the robust design of interplanetary trajectories in presence of severe disturbances. An open-source implementation of the state-of-the-art algorithm Proximal Policy Optimization is adopted. The resulting Guidance and Control Network provides both a robust nominal trajectory and the associated closed-loop guidance law.
  arXiv  Detail & Related papers  (2020-08-19T15:22:15Z)
• Guided Constrained Policy Optimization for Dynamic Quadrupedal Robot Locomotion [78.46388769788405]
  We introduce guided constrained policy optimization (GCPO), an RL framework based upon our implementation of constrained policy optimization (CPPO) We show that guided constrained RL offers faster convergence close to the desired optimum resulting in an optimal, yet physically feasible, robotic control behavior without the need for precise reward function tuning.
  arXiv  Detail & Related papers  (2020-02-22T10:15:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.