Online Monitoring of Object Detection Performance During Deployment

• URL: http://arxiv.org/abs/2011.07750v2
• Date: Tue, 9 Mar 2021 06:20:05 GMT
• Title: Online Monitoring of Object Detection Performance During Deployment
• Authors: Quazi Marufur Rahman, Niko S\"underhauf, Feras Dayoub
• Abstract summary: We introduce a cascaded neural network that monitors the performance of the object detector by predicting the quality of its mean average precision (mAP) on a sliding window of the input frames. We evaluate our proposed approach using different combinations of autonomous driving datasets and object detectors.
• Score: 6.166295570030645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: During deployment, an object detector is expected to operate at a similar performance level reported on its testing dataset. However, when deployed onboard mobile robots that operate under varying and complex environmental conditions, the detector's performance can fluctuate and occasionally degrade severely without warning. Undetected, this can lead the robot to take unsafe and risky actions based on low-quality and unreliable object detections. We address this problem and introduce a cascaded neural network that monitors the performance of the object detector by predicting the quality of its mean average precision (mAP) on a sliding window of the input frames. The proposed cascaded network exploits the internal features from the deep neural network of the object detector. We evaluate our proposed approach using different combinations of autonomous driving datasets and object detectors.

Related papers

• Uncertainty Estimation for 3D Object Detection via Evidential Learning [63.61283174146648]
  We introduce a framework for quantifying uncertainty in 3D object detection by leveraging an evidential learning loss on Bird's Eye View representations in the 3D detector. We demonstrate both the efficacy and importance of these uncertainty estimates on identifying out-of-distribution scenes, poorly localized objects, and missing (false negative) detections.
  arXiv  Detail & Related papers  (2024-10-31T13:13:32Z)
• Run-time Monitoring of 3D Object Detection in Automated Driving Systems Using Early Layer Neural Activation Patterns [12.384452095533396]
  Integrity monitoring of automated driving systems (ADS) is paramount for ensuring safety. Recent advancements in deep neural network (DNN)-based object detectors, their susceptibility to detection errors remains a significant concern.
  arXiv  Detail & Related papers  (2024-04-11T12:24:47Z)
• Run-time Introspection of 2D Object Detection in Automated Driving Systems Using Learning Representations [13.529124221397822]
  We introduce a novel introspection solution for 2D object detection based on Deep Neural Networks (DNNs) We implement several state-of-the-art (SOTA) introspection mechanisms for error detection in 2D object detection, using one-stage and two-stage object detectors evaluated on KITTI and BDD datasets. Our performance evaluation shows that the proposed introspection solution outperforms SOTA methods, achieving an absolute reduction in the missed error ratio of 9% to 17% in the BDD dataset.
  arXiv  Detail & Related papers  (2024-03-02T10:56:14Z)
• LS-VOS: Identifying Outliers in 3D Object Detections Using Latent Space Virtual Outlier Synthesis [10.920640666237833]
  LiDAR-based 3D object detectors have achieved unprecedented speed and accuracy in autonomous driving applications. They are often biased toward high-confidence predictions or return detections where no real object is present. We propose LS-VOS, a framework for identifying outliers in 3D object detections.
  arXiv  Detail & Related papers  (2023-10-02T07:44:26Z)
• Towards Building Self-Aware Object Detectors via Reliable Uncertainty Quantification and Calibration [17.461451218469062]
  In this work, we introduce the Self-Aware Object Detection (SAOD) task. The SAOD task respects and adheres to the challenges that object detectors face in safety-critical environments such as autonomous driving. We extensively use our framework, which introduces novel metrics and large scale test datasets, to test numerous object detectors.
  arXiv  Detail & Related papers  (2023-07-03T11:16:39Z)
• A Quality Index Metric and Method for Online Self-Assessment of Autonomous Vehicles Sensory Perception [164.93739293097605]
  We propose a novel evaluation metric, named as the detection quality index (DQI), which assesses the performance of camera-based object detection algorithms. We have developed a superpixel-based attention network (SPA-NET) that utilizes raw image pixels and superpixels as input to predict the proposed DQI evaluation metric.
  arXiv  Detail & Related papers  (2022-03-04T22:16:50Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• Robust Object Detection via Instance-Level Temporal Cycle Confusion [89.1027433760578]
  We study the effectiveness of auxiliary self-supervised tasks to improve the out-of-distribution generalization of object detectors. Inspired by the principle of maximum entropy, we introduce a novel self-supervised task, instance-level temporal cycle confusion (CycConf) For each object, the task is to find the most different object proposals in the adjacent frame in a video and then cycle back to itself for self-supervision.
  arXiv  Detail & Related papers  (2021-04-16T21:35:08Z)
• Robust and Accurate Object Detection via Adversarial Learning [111.36192453882195]
  This work augments the fine-tuning stage for object detectors by exploring adversarial examples. Our approach boosts the performance of state-of-the-art EfficientDets by +1.1 mAP on the object detection benchmark.
  arXiv  Detail & Related papers  (2021-03-23T19:45:26Z)
• Per-frame mAP Prediction for Continuous Performance Monitoring of Object Detection During Deployment [6.166295570030645]
  We propose an introspection approach to performance monitoring during deployment. We do so by predicting when the per-frame mean average precision drops below a critical threshold. We quantitatively evaluate and demonstrate our method's ability to reduce risk by trading off making an incorrect decision.
  arXiv  Detail & Related papers  (2020-09-18T06:37:52Z)
• Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance [95.86944752753564]
  We propose an online path planning architecture that extends the model predictive control (MPC) formulation to consider future location uncertainties. Our algorithm combines an object detection pipeline with a recurrent neural network (RNN) which infers the covariance of state estimates. The robustness of our methods is validated on complex quadruped robot dynamics and can be generally applied to most robotic platforms.
  arXiv  Detail & Related papers  (2020-07-28T07:34:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.