Model-Free versus Model-Based Reinforcement Learning for Fixed-Wing UAV Attitude Control Under Varying Wind Conditions

• URL: http://arxiv.org/abs/2409.17896v1
• Date: Thu, 26 Sep 2024 14:47:14 GMT
• Title: Model-Free versus Model-Based Reinforcement Learning for Fixed-Wing UAV Attitude Control Under Varying Wind Conditions
• Authors: David Olivares, Pierre Fournier, Pavan Vasishta, Julien Marzat
• Abstract summary: This paper evaluates and compares the performance of model-free and model-based reinforcement learning for the attitude control of fixed-wing unmanned aerial vehicles using PID as a reference point. Results show that the Temporal Difference Model Predictive Control agent outperforms both the PID controller and other model-free reinforcement learning methods in terms of tracking accuracy and robustness.
• Score: 1.474723404975345
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper evaluates and compares the performance of model-free and model-based reinforcement learning for the attitude control of fixed-wing unmanned aerial vehicles using PID as a reference point. The comparison focuses on their ability to handle varying flight dynamics and wind disturbances in a simulated environment. Our results show that the Temporal Difference Model Predictive Control agent outperforms both the PID controller and other model-free reinforcement learning methods in terms of tracking accuracy and robustness over different reference difficulties, particularly in nonlinear flight regimes. Furthermore, we introduce actuation fluctuation as a key metric to assess energy efficiency and actuator wear, and we test two different approaches from the literature: action variation penalty and conditioning for action policy smoothness. We also evaluate all control methods when subject to stochastic turbulence and gusts separately, so as to measure their effects on tracking performance, observe their limitations and outline their implications on the Markov decision process formalism.

Related papers

• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• A Reinforcement Learning Approach for Robust Supervisory Control of UAVs Under Disturbances [1.8799681615947088]
  We present an approach to supervisory reinforcement learning control for unmanned aerial vehicles (UAVs) We formulate a supervisory control architecture that interleaves with extant embedded control and demonstrates robustness to environmental disturbances in the form of adverse wind conditions.
  arXiv  Detail & Related papers  (2023-05-21T19:00:06Z)
• Model-Based Reinforcement Learning with Isolated Imaginations [61.67183143982074]
  We propose Iso-Dream++, a model-based reinforcement learning approach. We perform policy optimization based on the decoupled latent imaginations. This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild.
  arXiv  Detail & Related papers  (2023-03-27T02:55:56Z)
• Inferring Traffic Models in Terminal Airspace from Flight Tracks and Procedures [52.25258289718559]
  We propose a probabilistic model that can learn the variability from procedural data and flight tracks collected from radar surveillance data. We show how a pairwise model can be used to generate traffic involving an arbitrary number of aircraft.
  arXiv  Detail & Related papers  (2023-03-17T13:58:06Z)
• Incorporating Recurrent Reinforcement Learning into Model Predictive Control for Adaptive Control in Autonomous Driving [11.67417895998434]
  Model Predictive Control (MPC) is attracting tremendous attention in the autonomous driving task as a powerful control technique. In this paper, we reformulate the problem as a Partially Observed Markov Decision Process (POMDP) We then learn a recurrent policy continually adapting the parameters of the dynamics model via Recurrent Reinforcement Learning (RRL) for optimal and adaptive control.
  arXiv  Detail & Related papers  (2023-01-30T22:11:07Z)
• Interpretable Stochastic Model Predictive Control using Distributional Reinforced Estimation for Quadrotor Tracking Systems [0.8411385346896411]
  We present a novel trajectory tracker for autonomous quadrotor navigation in dynamic and complex environments. The proposed framework integrates a distributional Reinforcement Learning estimator for unknown aerodynamic effects into a Model Predictive Controller. We demonstrate our system to improve the cumulative tracking errors by at least 66% with unknown and diverse aerodynamic forces.
  arXiv  Detail & Related papers  (2022-05-14T23:27:38Z)
• Evaluating model-based planning and planner amortization for continuous control [79.49319308600228]
  We take a hybrid approach, combining model predictive control (MPC) with a learned model and model-free policy learning. We find that well-tuned model-free agents are strong baselines even for high DoF control problems. We show that it is possible to distil a model-based planner into a policy that amortizes the planning without any loss of performance.
  arXiv  Detail & Related papers  (2021-10-07T12:00:40Z)
• Reinforcement Learning with Formal Performance Metrics for Quadcopter Attitude Control under Non-nominal Contexts [2.198760145670348]
  We develop a robust form of a signal temporal logic to quantitatively evaluate the vehicle's behavior and measure the performance of controllers. We discuss the robustness of the obtained controllers, both to partial loss of power for one rotor and to wind gusts and finish by drawing conclusions on practical controller design by reinforcement learning.
  arXiv  Detail & Related papers  (2021-07-27T16:58:19Z)
• The Impact of Data on the Stability of Learning-Based Control- Extended Version [63.97366815968177]
  We propose a Lyapunov-based measure for quantifying the impact of data on the certifiable control performance. By modeling unknown system dynamics through Gaussian processes, we can determine the interrelation between model uncertainty and satisfaction of stability conditions.
  arXiv  Detail & Related papers  (2020-11-20T19:10:01Z)
• Comparison of Model Predictive and Reinforcement Learning Methods for Fault Tolerant Control [2.524528674141466]
  We present two adaptive fault-tolerant control schemes for a discrete time system based on hierarchical reinforcement learning. Experiments demonstrate that reinforcement learning-based controllers perform more robustly than model predictive controllers under faults, partially observable system models, and varying sensor noise levels.
  arXiv  Detail & Related papers  (2020-08-10T20:22:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.