Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis

• URL: http://arxiv.org/abs/2211.02736v4
• Date: Wed, 10 Apr 2024 04:51:33 GMT
• Title: Discovering Closed-Loop Failures of Vision-Based Controllers via Reachability Analysis
• Authors: Kaustav Chakraborty, Somil Bansal,
• Abstract summary: Machine learning driven image-based controllers allow robotic systems to take intelligent actions based on the visual feedback from their environment. Existing methods leverage simulation-based testing (or falsification) to find the failures of vision-based controllers. In this work, we cast the problem of finding closed-loop vision failures as a Hamilton-Jacobi (HJ) reachability problem.
• Score: 7.679478106628509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning driven image-based controllers allow robotic systems to take intelligent actions based on the visual feedback from their environment. Understanding when these controllers might lead to system safety violations is important for their integration in safety-critical applications and engineering corrective safety measures for the system. Existing methods leverage simulation-based testing (or falsification) to find the failures of vision-based controllers, i.e., the visual inputs that lead to closed-loop safety violations. However, these techniques do not scale well to the scenarios involving high-dimensional and complex visual inputs, such as RGB images. In this work, we cast the problem of finding closed-loop vision failures as a Hamilton-Jacobi (HJ) reachability problem. Our approach blends simulation-based analysis with HJ reachability methods to compute an approximation of the backward reachable tube (BRT) of the system, i.e., the set of unsafe states for the system under vision-based controllers. Utilizing the BRT, we can tractably and systematically find the system states and corresponding visual inputs that lead to closed-loop failures. These visual inputs can be subsequently analyzed to find the input characteristics that might have caused the failure. Besides its scalability to high-dimensional visual inputs, an explicit computation of BRT allows the proposed approach to capture non-trivial system failures that are difficult to expose via random simulations. We demonstrate our framework on two case studies involving an RGB image-based neural network controller for (a) autonomous indoor navigation, and (b) autonomous aircraft taxiing.

Related papers

• Scalable Surrogate Verification of Image-based Neural Network Control Systems using Composition and Unrolling [9.633494094538017]
  We build on work that considers a surrogate verification approach, training a conditional generative adversarial network (cGAN) as an image generator in place of the real world. We overcome one-step error by composing the system's dynamics along with the cGAN and neural network controller. We reduce multi-step error by repeating the single-step composition, essentially unrolling multiple steps of the control loop into a large neural network.
  arXiv  Detail & Related papers  (2024-05-28T19:56:53Z)
• Floor extraction and door detection for visually impaired guidance [78.94595951597344]
  Finding obstacle-free paths in unknown environments is a big navigation issue for visually impaired people and autonomous robots. New devices based on computer vision systems can help impaired people to overcome the difficulties of navigating in unknown environments in safe conditions. In this work it is proposed a combination of sensors and algorithms that can lead to the building of a navigation system for visually impaired people.
  arXiv  Detail & Related papers  (2024-01-30T14:38:43Z)
• Detecting and Mitigating System-Level Anomalies of Vision-Based Controllers [7.095058159492494]
  Vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. In this work, we introduce a run-time anomaly monitor to detect and mitigate such closed-loop, system-level failures. We validate the proposed approach on an autonomous aircraft taxiing system that uses a vision-based controller for taxiing.
  arXiv  Detail & Related papers  (2023-09-23T20:33:38Z)
• In-Distribution Barrier Functions: Self-Supervised Policy Filters that Avoid Out-of-Distribution States [84.24300005271185]
  We propose a control filter that wraps any reference policy and effectively encourages the system to stay in-distribution with respect to offline-collected safe demonstrations. Our method is effective for two different visuomotor control tasks in simulation environments, including both top-down and egocentric view settings.
  arXiv  Detail & Related papers  (2023-01-27T22:28:19Z)
• Testing Rare Downstream Safety Violations via Upstream Adaptive Sampling of Perception Error Models [20.815131169609316]
  This paper combines perception error models -- surrogates for a sensor-based detection system -- with state-dependent adaptive importance sampling. Experiments with an autonomous braking system equipped with an RGB obstacle-detector show that our method can calculate accurate failure probabilities.
  arXiv  Detail & Related papers  (2022-09-20T12:26:06Z)
• Recursively Feasible Probabilistic Safe Online Learning with Control Barrier Functions [60.26921219698514]
  We introduce a model-uncertainty-aware reformulation of CBF-based safety-critical controllers. We then present the pointwise feasibility conditions of the resulting safety controller. We use these conditions to devise an event-triggered online data collection strategy.
  arXiv  Detail & Related papers  (2022-08-23T05:02:09Z)
• Monitoring of Perception Systems: Deterministic, Probabilistic, and Learning-based Fault Detection and Identification [21.25149064251918]
  We formalize the problem of runtime fault detection and identification in perception systems. We provide a set of deterministic, probabilistic, and learning-based algorithms that use diagnostic graphs to perform fault detection and identification. We conclude the paper with an experimental evaluation, which recreates several realistic failure modes in the LGSVL open-source autonomous driving simulator.
  arXiv  Detail & Related papers  (2022-05-22T19:08:45Z)
• Learning Robust Output Control Barrier Functions from Safe Expert Demonstrations [50.37808220291108]
  This paper addresses learning safe output feedback control laws from partial observations of expert demonstrations. We first propose robust output control barrier functions (ROCBFs) as a means to guarantee safety. We then formulate an optimization problem to learn ROCBFs from expert demonstrations that exhibit safe system behavior.
  arXiv  Detail & Related papers  (2021-11-18T23:21:00Z)
• Monitoring and Diagnosability of Perception Systems [21.25149064251918]
  We propose a mathematical model for runtime monitoring and fault detection and identification in perception systems. We demonstrate our monitoring system, dubbed PerSyS, in realistic simulations using the LGSVL self-driving simulator and the Apollo Auto autonomy software stack.
  arXiv  Detail & Related papers  (2020-11-11T23:03:14Z)
• Learning Hybrid Control Barrier Functions from Data [66.37785052099423]
  Motivated by the lack of systematic tools to obtain safe control laws for hybrid systems, we propose an optimization-based framework for learning certifiably safe control laws from data. In particular, we assume a setting in which the system dynamics are known and in which data exhibiting safe system behavior is available.
  arXiv  Detail & Related papers  (2020-11-08T23:55:02Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.