CTSAC: Curriculum-Based Transformer Soft Actor-Critic for Goal-Oriented Robot Exploration

• URL: http://arxiv.org/abs/2503.14254v1
• Date: Tue, 18 Mar 2025 13:44:29 GMT
• Title: CTSAC: Curriculum-Based Transformer Soft Actor-Critic for Goal-Oriented Robot Exploration
• Authors: Chunyu Yang, Shengben Bi, Yihui Xu, Xin Zhang,
• Abstract summary: Reinforcement Learning (RL) is becoming a promising approach in the field of autonomous robotic exploration.<n>Current RL-based exploration algorithms often face limited environmental reasoning capabilities, slow convergence rates, and substantial challenges in Sim-To-Real transfer.<n>We propose a Curriculum Learning-based Transformer Reinforcement Learning Algorithm (CTSAC) aimed at improving both exploration efficiency and transfer performance.
• Score: 3.435901586870572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the increasing demand for efficient and flexible robotic exploration solutions, Reinforcement Learning (RL) is becoming a promising approach in the field of autonomous robotic exploration. However, current RL-based exploration algorithms often face limited environmental reasoning capabilities, slow convergence rates, and substantial challenges in Sim-To-Real (S2R) transfer. To address these issues, we propose a Curriculum Learning-based Transformer Reinforcement Learning Algorithm (CTSAC) aimed at improving both exploration efficiency and transfer performance. To enhance the robot's reasoning ability, a Transformer is integrated into the perception network of the Soft Actor-Critic (SAC) framework, leveraging historical information to improve the farsightedness of the strategy. A periodic review-based curriculum learning is proposed, which enhances training efficiency while mitigating catastrophic forgetting during curriculum transitions. Training is conducted on the ROS-Gazebo continuous robotic simulation platform, with LiDAR clustering optimization to further reduce the S2R gap. Experimental results demonstrate the CTSAC algorithm outperforms the state-of-the-art non-learning and learning-based algorithms in terms of success rate and success rate-weighted exploration time. Moreover, real-world experiments validate the strong S2R transfer capabilities of CTSAC.

Related papers

• Digital Twin-Enabled Real-Time Control in Robotic Additive Manufacturing via Soft Actor-Critic Reinforcement Learning [2.5709786140685633]
  This research presents a novel approach integrating Soft Actor-Critic (SAC) reinforcement learning with digital twin technology.<n>We demonstrate our methodology using a Viper X300s robot arm, implementing two distinct control scenarios.<n>Results show rapid policy convergence and robust task execution in both simulated and physical environments.
  arXiv  Detail & Related papers  (2025-01-29T22:06:53Z)
• Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning [47.785786984974855]
  We present a human-in-the-loop vision-based RL system that demonstrates impressive performance on a diverse set of dexterous manipulation tasks. Our approach integrates demonstrations and human corrections, efficient RL algorithms, and other system-level design choices to learn policies. We show that our method significantly outperforms imitation learning baselines and prior RL approaches, with an average 2x improvement in success rate and 1.8x faster execution.
  arXiv  Detail & Related papers  (2024-10-29T08:12:20Z)
• Learn to Teach: Sample-Efficient Privileged Learning for Humanoid Locomotion over Diverse Terrains [6.967583364984562]
  This work proposes a novel one-stage training framework-Learn to Teach (L2T)-which unifies teacher and student policy learning.<n>Our approach recycles simulator samples and synchronizes the learning trajectories through shared dynamics, significantly reducing sample complexities and training time.<n>We validate the RL variant (L2T-RL) through extensive simulations and hardware tests on the Digit robot, demonstrating zero-shot sim-to-real transfer and robust performance over 12+ challenging terrains without depth estimation modules.
  arXiv  Detail & Related papers  (2024-02-09T21:16:43Z)
• SERL: A Software Suite for Sample-Efficient Robotic Reinforcement Learning [85.21378553454672]
  We develop a library containing a sample efficient off-policy deep RL method, together with methods for computing rewards and resetting the environment. We find that our implementation can achieve very efficient learning, acquiring policies for PCB board assembly, cable routing, and object relocation. These policies achieve perfect or near-perfect success rates, extreme robustness even under perturbations, and exhibit emergent robustness recovery and correction behaviors.
  arXiv  Detail & Related papers  (2024-01-29T10:01:10Z)
• Back-stepping Experience Replay with Application to Model-free Reinforcement Learning for a Soft Snake Robot [15.005962159112002]
  Back-stepping Experience Replay (BER) is compatible with arbitrary off-policy reinforcement learning algorithms. We present an application of BER in a model-free RL approach for the locomotion and navigation of a soft snake robot.
  arXiv  Detail & Related papers  (2024-01-21T02:17:16Z)
• Tactile Active Inference Reinforcement Learning for Efficient Robotic Manipulation Skill Acquisition [10.072992621244042]
  We propose a novel method for skill learning in robotic manipulation called Tactile Active Inference Reinforcement Learning (Tactile-AIRL) To enhance the performance of reinforcement learning (RL), we introduce active inference, which integrates model-based techniques and intrinsic curiosity into the RL process. We demonstrate that our method achieves significantly high training efficiency in non-prehensile objects pushing tasks.
  arXiv  Detail & Related papers  (2023-11-19T10:19:22Z)
• Confidence-Controlled Exploration: Efficient Sparse-Reward Policy Learning for Robot Navigation [72.24964965882783]
  Reinforcement learning (RL) is a promising approach for robotic navigation, allowing robots to learn through trial and error. Real-world robotic tasks often suffer from sparse rewards, leading to inefficient exploration and suboptimal policies. We introduce Confidence-Controlled Exploration (CCE), a novel method that improves sample efficiency in RL-based robotic navigation without modifying the reward function.
  arXiv  Detail & Related papers  (2023-06-09T18:45:15Z)
• MARLIN: Soft Actor-Critic based Reinforcement Learning for Congestion Control in Real Networks [63.24965775030673]
  We propose a novel Reinforcement Learning (RL) approach to design generic Congestion Control (CC) algorithms. Our solution, MARLIN, uses the Soft Actor-Critic algorithm to maximize both entropy and return. We trained MARLIN on a real network with varying background traffic patterns to overcome the sim-to-real mismatch.
  arXiv  Detail & Related papers  (2023-02-02T18:27:20Z)
• PI-ARS: Accelerating Evolution-Learned Visual-Locomotion with Predictive Information Representations [32.37414300338581]
  Evolution Strategy (ES) algorithms have shown promising results in training complex robotic control policies. PI-ARS combines a gradient-based representation learning technique, Predictive Information (PI), with a gradient-free ES algorithm, Augmented Random Search (ARS) We show PI-ARS demonstrates significantly better learning efficiency and performance compared to the ARS baseline.
  arXiv  Detail & Related papers  (2022-07-27T00:26:15Z)
• Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives [92.0321404272942]
  Reinforcement learning can be used to build general-purpose robotic systems. However, training RL agents to solve robotics tasks still remains challenging. In this work, we manually specify a library of robot action primitives (RAPS), parameterized with arguments that are learned by an RL policy. We find that our simple change to the action interface substantially improves both the learning efficiency and task performance.
  arXiv  Detail & Related papers  (2021-10-28T17:59:30Z)
• RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Real [74.45688231140689]
  We introduce the RL-scene consistency loss for image translation, which ensures that the translation operation is invariant with respect to the Q-values associated with the image. We obtain RL-CycleGAN, a new approach for simulation-to-real-world transfer for reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T08:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.