When Uncertainty Leads to Unsafety: Empirical Insights into the Role of Uncertainty in Unmanned Aerial Vehicle Safety

• URL: http://arxiv.org/abs/2501.08908v2
• Date: Tue, 10 Jun 2025 14:36:19 GMT
• Title: When Uncertainty Leads to Unsafety: Empirical Insights into the Role of Uncertainty in Unmanned Aerial Vehicle Safety
• Abstract summary: This work investigates the relationship between the behavioral uncertainty of a UAV and the unsafety of its flight.<n>By quantifying uncertainty, it is possible to develop a predictor for unsafety, which acts as a flight supervisor.<n>Superialist achieved high performance in detecting uncertain behaviors with up to 96% precision and 93% recall.
• Score: 9.432686404777584
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the recent developments in obstacle avoidance and other safety features, autonomous Unmanned Aerial Vehicles (UAVs) continue to face safety challenges. No previous work investigated the relationship between the behavioral uncertainty of a UAV, characterized in this work by inconsistent or erratic control signal patterns, and the unsafety of its flight. By quantifying uncertainty, it is possible to develop a predictor for unsafety, which acts as a flight supervisor. We conducted a large-scale empirical investigation of safety violations using PX4-Autopilot, an open-source UAV software platform. Our dataset of over 5,000 simulated flights, created to challenge obstacle avoidance, allowed us to explore the relation between uncertain UAV decisions and safety violations: up to 89% of unsafe UAV states exhibit significant decision uncertainty, and up to 74% of uncertain decisions lead to unsafe states. Based on these findings, we implemented Superialist (Supervising Autonomous Aerial Vehicles), a runtime uncertainty detector based on autoencoders, the state-of-the-art technology for anomaly detection. Superialist achieved high performance in detecting uncertain behaviors with up to 96% precision and 93% recall. Despite the observed performance degradation when using the same approach for predicting unsafety (up to 74% precision and 87% recall), Superialist enabled early prediction of unsafe states up to 50 seconds in advance.

Related papers

• On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment [54.30690671490447]
  Existing safety-alignment signals are largely response-level or off-policy.<n>FATE transforms verifier-scored failures into repair supervision without expert demonstrations.<n>FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%.
  arXiv  Detail & Related papers  (2026-05-12T09:56:28Z)
• SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do [0.0]
  SafetyDrift models agent safety trajectories as absorbing Markov chains.<n>In communication tasks, agents that reach even a mild risk state have an 85% chance of violating safety within five steps.<n>A monitor built on these models detects 94.7% of violations with 3.7 steps of advance warning at negligible computational cost.
  arXiv  Detail & Related papers  (2026-03-28T05:52:04Z)
• Conformal Reachability for Safe Control in Unknown Environments [29.315278038378835]
  We develop a probabilistic verification framework for unknown dynamical systems.<n>We use conformal prediction to obtain valid uncertainty intervals for the unknown dynamics at each time step.<n>We also develop an algorithmic approach for training control policies that optimize nominal reward while also maximizing the planning horizon.
  arXiv  Detail & Related papers  (2026-02-03T18:01:38Z)
• Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning [7.0366838632617705]
  We develop a data-driven approach for safety monitoring in flight testing, where pilots perform maneuvers on aircraft with uncertain parameters.<n>We use offline trajectory simulation to learn a calibrated statistical model of the short-term safety risk facing pilots.<n>We evaluate our method on a flight dynamics model with uncertain parameters, demonstrating its ability to reliably identify unsafe scenarios.
  arXiv  Detail & Related papers  (2025-11-25T19:57:07Z)
• Rethinking Safety in LLM Fine-tuning: An Optimization Perspective [56.31306558218838]
  We show that poor optimization choices, rather than inherent trade-offs, often cause safety problems, measured as harmful responses to adversarial prompts.<n>We propose a simple exponential moving average (EMA) momentum technique in parameter space that preserves safety performance.<n>Our experiments on the Llama families across multiple datasets demonstrate that safety problems can largely be avoided without specialized interventions.
  arXiv  Detail & Related papers  (2025-08-17T23:46:36Z)
• Efficient Safety Testing of Autonomous Vehicles via Adaptive Search over Crash-Derived Scenarios [6.000851091286008]
  This study focuses on designing an accelerated testing algorithm for AVs in safety-critical scenarios.<n>Baidu Apollo, an advanced black-box automated driving system (ADS) is integrated to control the behavior of the ego vehicle.<n> Experimental results demonstrate a significant enhancement in testing efficiency when utilizing ALVNS-SA.
  arXiv  Detail & Related papers  (2025-08-07T13:55:01Z)
• OCCUQ: Exploring Efficient Uncertainty Quantification for 3D Occupancy Prediction [9.742801351723482]
  Current methods often overlook uncertainties arising from adversarial conditions or distributional shifts. We propose an efficient adaptation of an uncertainty estimation technique for 3D occupancy prediction. Our approach consistently demonstrates reliable uncertainty measures, indicating its potential for enhancing the robustness of autonomous driving systems.
  arXiv  Detail & Related papers  (2025-03-13T17:50:07Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress? [59.96471873997733]
  We propose an empirical foundation for developing more meaningful safety metrics and define AI safety in a machine learning research context. We aim to provide a more rigorous framework for AI safety research, advancing the science of safety evaluations and clarifying the path towards measurable progress.
  arXiv  Detail & Related papers  (2024-07-31T17:59:24Z)
• Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning [0.0]
  We propose a safe reinforcement learning (RL) approach that utilizes an anomalous state sequence to enhance RL safety. In experiments on multiple safety-critical environments including self-driving cars, our solution approach successfully learns safer policies.
  arXiv  Detail & Related papers  (2024-07-29T10:30:07Z)
• Predicting Safety Misbehaviours in Autonomous Driving Systems using Uncertainty Quantification [8.213390074932132]
  This paper evaluates different uncertainty quantification methods from the deep learning domain for the anticipatory testing of safety-critical misbehaviours. We compute uncertainty scores as the vehicle executes, following the intuition that high uncertainty scores are indicative of unsupported runtime conditions. In our study, we conducted an evaluation of the effectiveness and computational overhead associated with two uncertainty quantification methods, namely MC- Dropout and Deep Ensembles, for misbehaviour avoidance.
  arXiv  Detail & Related papers  (2024-04-29T10:28:28Z)
• ACAV: A Framework for Automatic Causality Analysis in Autonomous Vehicle Accident Recordings [5.578446693797519]
  Recent fatalities have emphasized the importance of safety validation through large-scale testing. We propose ACAV, an automated framework designed to conduct causality analysis for AV accident recordings. We evaluate ACAV on the Apollo ADS, finding that it can identify five distinct types of causal events in 93.64% of 110 accident recordings.
  arXiv  Detail & Related papers  (2024-01-13T12:41:05Z)
• KnowSafe: Combined Knowledge and Data Driven Hazard Mitigation in Artificial Pancreas Systems [3.146076597280736]
  KnowSafe predicts and mitigates safety hazards resulting from safety-critical malicious attacks or accidental faults targeting a CPS controller. We integrate domain-specific knowledge of safety constraints and context-specific mitigation actions with machine learning (ML) techniques. KnowSafe outperforms the state-of-the-art by achieving higher accuracy in predicting system state trajectories and potential hazards.
  arXiv  Detail & Related papers  (2023-11-13T16:43:34Z)
• A Counterfactual Safety Margin Perspective on the Scoring of Autonomous Vehicles' Riskiness [52.27309191283943]
  This paper presents a data-driven framework for assessing the risk of different AVs' behaviors. We propose the notion of counterfactual safety margin, which represents the minimum deviation from nominal behavior that could cause a collision.
  arXiv  Detail & Related papers  (2023-08-02T09:48:08Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Meta-Learning Priors for Safe Bayesian Optimization [72.8349503901712]
  We build on a meta-learning algorithm, F-PACOH, capable of providing reliable uncertainty quantification in settings of data scarcity. As core contribution, we develop a novel framework for choosing safety-compliant priors in a data-riven manner. On benchmark functions and a high-precision motion system, we demonstrate that our meta-learned priors accelerate the convergence of safe BO approaches.
  arXiv  Detail & Related papers  (2022-10-03T08:38:38Z)
• Bayesian autoencoders with uncertainty quantification: Towards trustworthy anomaly detection [78.24964622317634]
  In this work, the formulation of Bayesian autoencoders (BAEs) is adopted to quantify the total anomaly uncertainty. To evaluate the quality of uncertainty, we consider the task of classifying anomalies with the additional option of rejecting predictions of high uncertainty. Our experiments demonstrate the effectiveness of the BAE and total anomaly uncertainty on a set of benchmark datasets and two real datasets for manufacturing.
  arXiv  Detail & Related papers  (2022-02-25T12:20:04Z)
• Conservative Safety Critics for Exploration [120.73241848565449]
  We study the problem of safe exploration in reinforcement learning (RL) We learn a conservative safety estimate of environment states through a critic. We show that the proposed approach can achieve competitive task performance while incurring significantly lower catastrophic failure rates.
  arXiv  Detail & Related papers  (2020-10-27T17:54:25Z)
• A Comparison of Uncertainty Estimation Approaches in Deep Learning Components for Autonomous Vehicle Applications [0.0]
  Key factor for ensuring safety in Autonomous Vehicles (AVs) is to avoid any abnormal behaviors under undesirable and unpredicted circumstances. Different methods for uncertainty quantification have recently been proposed to measure the inevitable source of errors in data and models. These methods require a higher computational load, a higher memory footprint, and introduce extra latency, which can be prohibitive in safety-critical applications.
  arXiv  Detail & Related papers  (2020-06-26T18:55:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.