Curriculum-based Sample Efficient Reinforcement Learning for Robust Stabilization of a Quadrotor

• URL: http://arxiv.org/abs/2501.18490v1
• Date: Thu, 30 Jan 2025 17:05:32 GMT
• Title: Curriculum-based Sample Efficient Reinforcement Learning for Robust Stabilization of a Quadrotor
• Authors: Fausto Mauricio Lagos Suarez, Akshit Saradagi, Vidya Sumathy, Shruti Kotpaliwar, George Nikolakopoulos,
• Abstract summary: This article introduces a curriculum learning approach to develop a robust stabilizing controller for a Quadrotor.<n>The learning objective is to achieve desired positions from random initial conditions.<n>A novel additive reward function is proposed, to incorporate transient and steady-state performance specifications.
• Score: 3.932152385564876
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This article introduces a curriculum learning approach to develop a reinforcement learning-based robust stabilizing controller for a Quadrotor that meets predefined performance criteria. The learning objective is to achieve desired positions from random initial conditions while adhering to both transient and steady-state performance specifications. This objective is challenging for conventional one-stage end-to-end reinforcement learning, due to the strong coupling between position and orientation dynamics, the complexity in designing and tuning the reward function, and poor sample efficiency, which necessitates substantial computational resources and leads to extended convergence times. To address these challenges, this work decomposes the learning objective into a three-stage curriculum that incrementally increases task complexity. The curriculum begins with learning to achieve stable hovering from a fixed initial condition, followed by progressively introducing randomization in initial positions, orientations and velocities. A novel additive reward function is proposed, to incorporate transient and steady-state performance specifications. The results demonstrate that the Proximal Policy Optimization (PPO)-based curriculum learning approach, coupled with the proposed reward structure, achieves superior performance compared to a single-stage PPO-trained policy with the same reward function, while significantly reducing computational resource requirements and convergence time. The curriculum-trained policy's performance and robustness are thoroughly validated under random initial conditions and in the presence of disturbances.

Related papers

• COPO: Consistency-Aware Policy Optimization [17.328515578426227]
  Reinforcement learning has significantly enhanced the reasoning capabilities of Large Language Models (LLMs) in complex problem-solving tasks.<n>Recently, the introduction of DeepSeek R1 has inspired a surge of interest in leveraging rule-based rewards as a low-cost alternative for computing advantage functions and guiding policy optimization.<n>We propose a consistency-aware policy optimization framework that introduces a structured global reward based on outcome consistency.
  arXiv  Detail & Related papers  (2025-08-06T07:05:18Z)
• Test-time Offline Reinforcement Learning on Goal-related Experience [50.94457794664909]
  Research in foundation models has shown that performance can be substantially improved through test-time training.<n>We propose a novel self-supervised data selection criterion, which selects transitions from an offline dataset according to their relevance to the current state.<n>Our goal-conditioned test-time training (GC-TTT) algorithm applies this routine in a receding-horizon fashion during evaluation, adapting the policy to the current trajectory as it is being rolled out.
  arXiv  Detail & Related papers  (2025-07-24T21:11:39Z)
• Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments [78.15330971155778]
  Posterior and Diversity Synergized Task Sampling (PDTS) is an easy-to-implement method to accommodate fast and robust sequential decision-making. PDTS unlocks the potential of robust active task sampling, significantly improves the zero-shot and few-shot adaptation robustness in challenging tasks, and even accelerates the learning process under certain scenarios.
  arXiv  Detail & Related papers  (2025-04-27T07:27:17Z)
• Closing the Intent-to-Behavior Gap via Fulfillment Priority Logic [1.4542411354617986]
  This paper presents the concept of objective fulfillment upon which we build Fulfillment Priority Logic (FPL) Our novel Balanced Policy Gradient algorithm leverages FPL specifications to achieve up to 500% better sample efficiency compared to Soft Actor Critic.
  arXiv  Detail & Related papers  (2025-03-04T18:45:20Z)
• Model Predictive Task Sampling for Efficient and Robust Adaptation [46.92143725900031]
  We introduce Model Predictive Task Sampling (MPTS), a framework that bridges the task space and adaptation risk landscape. MPTS employs a generative model to characterize the episodic optimization process and predicts task-specific adaptation risk via posterior inference. MPTS seamlessly integrates into zero-shot, few-shot, and supervised finetuning settings.
  arXiv  Detail & Related papers  (2025-01-19T13:14:53Z)
• Adaptive Reward Design for Reinforcement Learning [2.3031174164121127]
  We propose a suite of reward functions that incentivize an RL agent to complete a task specified by a formula as much as possible.<n>We develop an adaptive reward shaping approach that dynamically updates reward functions during the learning process.
  arXiv  Detail & Related papers  (2024-12-14T18:04:18Z)
• Rethinking Inverse Reinforcement Learning: from Data Alignment to Task Alignment [7.477559660351106]
  imitation learning (IL) algorithms use inverse reinforcement learning (IRL) to infer a reward function that aligns with a demonstration. We propose a novel framework for IRL-based IL that prioritizes task alignment over conventional data alignment.
  arXiv  Detail & Related papers  (2024-10-31T07:08:14Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Directly Attention Loss Adjusted Prioritized Experience Replay [0.07366405857677226]
  Prioritized Replay Experience (PER) enables the model to learn more about relatively important samples by artificially changing their accessed frequencies. DALAP is proposed, which can directly quantify the changed extent of the shifted distribution through Parallel Self-Attention network.
  arXiv  Detail & Related papers  (2023-11-24T10:14:05Z)
• Time-series Generation by Contrastive Imitation [87.51882102248395]
  We study a generative framework that seeks to combine the strengths of both: Motivated by a moment-matching objective to mitigate compounding error, we optimize a local (but forward-looking) transition policy. At inference, the learned policy serves as the generator for iterative sampling, and the learned energy serves as a trajectory-level measure for evaluating sample quality.
  arXiv  Detail & Related papers  (2023-11-02T16:45:25Z)
• When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning [62.00672284480755]
  This paper aims to recover the structure of rewards and environment dynamics that underlie observed actions in a fixed, finite set of demonstrations from an expert agent. Accurate models of expertise in executing a task has applications in safety-sensitive applications such as clinical decision making and autonomous driving.
  arXiv  Detail & Related papers  (2023-02-15T04:14:20Z)
• Off-Policy Reinforcement Learning with Delayed Rewards [16.914712720033524]
  In many real-world tasks, instant rewards are not readily accessible or defined immediately after the agent performs actions. In this work, we first formally define the environment with delayed rewards and discuss the challenges raised due to the non-Markovian nature of such environments. We introduce a general off-policy RL framework with a new Q-function formulation that can handle the delayed rewards with theoretical convergence guarantees.
  arXiv  Detail & Related papers  (2021-06-22T15:19:48Z)
• Efficient Empowerment Estimation for Unsupervised Stabilization [75.32013242448151]
  empowerment principle enables unsupervised stabilization of dynamical systems at upright positions. We propose an alternative solution based on a trainable representation of a dynamical system as a Gaussian channel. We show that our method has a lower sample complexity, is more stable in training, possesses the essential properties of the empowerment function, and allows estimation of empowerment from images.
  arXiv  Detail & Related papers  (2020-07-14T21:10:16Z)
• Temporal-Logic-Based Reward Shaping for Continuing Learning Tasks [57.17673320237597]
  In continuing tasks, average-reward reinforcement learning may be a more appropriate problem formulation than the more common discounted reward formulation. This paper presents the first reward shaping framework for average-reward learning. It proves that, under standard assumptions, the optimal policy under the original reward function can be recovered.
  arXiv  Detail & Related papers  (2020-07-03T05:06:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.