Towards Cooperative Flight Control Using Visual-Attention

• URL: http://arxiv.org/abs/2212.11084v2
• Date: Wed, 20 Sep 2023 20:50:46 GMT
• Title: Towards Cooperative Flight Control Using Visual-Attention
• Authors: Lianhao Yin, Makram Chahine, Tsun-Hsuan Wang, Tim Seyde, Chao Liu, Mathias Lechner, Ramin Hasani, Daniela Rus
• Abstract summary: We propose a vision-based air-guardian system to enable parallel autonomy between a pilot and a control system. Our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention.
• Score: 61.99121057062421
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The cooperation of a human pilot with an autonomous agent during flight control realizes parallel autonomy. We propose an air-guardian system that facilitates cooperation between a pilot with eye tracking and a parallel end-to-end neural control system. Our vision-based air-guardian system combines a causal continuous-depth neural network model with a cooperation layer to enable parallel autonomy between a pilot and a control system based on perceived differences in their attention profiles. The attention profiles for neural networks are obtained by computing the networks' saliency maps (feature importance) through the VisualBackProp algorithm, while the attention profiles for humans are either obtained by eye tracking of human pilots or saliency maps of networks trained to imitate human pilots. When the attention profile of the pilot and guardian agents align, the pilot makes control decisions. Otherwise, the air-guardian makes interventions and takes over the control of the aircraft. We show that our attention-based air-guardian system can balance the trade-off between its level of involvement in the flight and the pilot's expertise and attention. The guardian system is particularly effective in situations where the pilot was distracted due to information overload. We demonstrate the effectiveness of our method for navigating flight scenarios in simulation with a fixed-wing aircraft and on hardware with a quadrotor platform.

Related papers

• Estimating See and Be Seen Performance with an Airborne Visual Acquisition Model [0.0]
  Separation provision and collision avoidance are fundamental components of layered conflict management system. Pilots have visual-based separation responsibilities to see and be seen to maintain separation between aircraft. Drone interactions with crewed aircraft should not be more hazardous than interactions between traditional aviation aircraft.
  arXiv  Detail & Related papers  (2023-06-29T11:39:10Z)
• Learning Deep Sensorimotor Policies for Vision-based Autonomous Drone Racing [52.50284630866713]
  Existing systems often require hand-engineered components for state estimation, planning, and control. This paper tackles the vision-based autonomous-drone-racing problem by learning deep sensorimotor policies.
  arXiv  Detail & Related papers  (2022-10-26T19:03:17Z)
• The eyes and hearts of UAV pilots: observations of physiological responses in real-life scenarios [64.0476282000118]
  In civil and military aviation, pilots can train themselves on realistic simulators to tune their reaction and reflexes. This work aims to provide a solution to gather pilots behavior out in the field and help them increase their performance.
  arXiv  Detail & Related papers  (2022-10-26T14:16:56Z)
• The State of Aerial Surveillance: A Survey [62.198765910573556]
  This paper provides a comprehensive overview of human-centric aerial surveillance tasks from a computer vision and pattern recognition perspective. The main object of interest is humans, where single or multiple subjects are to be detected, identified, tracked, re-identified and have their behavior analyzed.
  arXiv  Detail & Related papers  (2022-01-09T20:13:27Z)
• Visual Attention Prediction Improves Performance of Autonomous Drone Racing Agents [45.36060508554703]
  Humans race drones faster than neural networks trained for end-to-end autonomous flight. This work investigates whether neural networks capable of imitating human eye gaze behavior and attention can improve neural network performance.
  arXiv  Detail & Related papers  (2022-01-07T18:07:51Z)
• Evolved neuromorphic radar-based altitude controller for an autonomous open-source blimp [4.350434044677268]
  In this paper, we propose an evolved altitude controller based on an SNN for a robotic airship. We also present an SNN-based controller architecture, an evolutionary framework for training the network in a simulated environment, and a control strategy for ameliorating the gap with reality.
  arXiv  Detail & Related papers  (2021-10-01T20:48:43Z)
• Scalable Perception-Action-Communication Loops with Convolutional and Graph Neural Networks [208.15591625749272]
  We present a perception-action-communication loop design using Vision-based Graph Aggregation and Inference (VGAI) Our framework is implemented by a cascade of a convolutional and a graph neural network (CNN / GNN), addressing agent-level visual perception and feature learning. We demonstrate that VGAI yields performance comparable to or better than other decentralized controllers.
  arXiv  Detail & Related papers  (2021-06-24T23:57:21Z)
• Deep4Air: A Novel Deep Learning Framework for Airport Airside Surveillance [0.9449650062296822]
  We propose a novel computer-vision based framework, namely "Deep4Air" It provides real-time speed and distance analytics for aircraft on runways and taxiways. The proposed framework includes an adaptive deep neural network for efficiently detecting and tracking aircraft.
  arXiv  Detail & Related papers  (2020-10-02T06:33:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.