On the Black-box Explainability of Object Detection Models for Safe and Trustworthy Industrial Applications

• URL: http://arxiv.org/abs/2411.00818v2
• Date: Thu, 28 Nov 2024 08:09:26 GMT
• Title: On the Black-box Explainability of Object Detection Models for Safe and Trustworthy Industrial Applications
• Authors: Alain Andres, Aitor Martinez-Seras, Ibai Laña, Javier Del Ser,
• Abstract summary: We focus on model-agnostic explainability methods for object detection models and propose D-P, an extension of the Morphological Fragmental Perturbation Pyramid (P) technique to generate explanations. We evaluate these methods on real-world industrial and robotic datasets, examining the influence of parameters such as the number of masks, model size, and image resolution on the quality of explanations.
• Score: 7.848637922112521
• License:
• Abstract: In the realm of human-machine interaction, artificial intelligence has become a powerful tool for accelerating data modeling tasks. Object detection methods have achieved outstanding results and are widely used in critical domains like autonomous driving and video surveillance. However, their adoption in high-risk applications, where errors may cause severe consequences, remains limited. Explainable Artificial Intelligence methods aim to address this issue, but many existing techniques are model-specific and designed for classification tasks, making them less effective for object detection and difficult for non-specialists to interpret. In this work we focus on model-agnostic explainability methods for object detection models and propose D-MFPP, an extension of the Morphological Fragmental Perturbation Pyramid (MFPP) technique based on segmentation-based masks to generate explanations. Additionally, we introduce D-Deletion, a novel metric combining faithfulness and localization, adapted specifically to meet the unique demands of object detectors. We evaluate these methods on real-world industrial and robotic datasets, examining the influence of parameters such as the number of masks, model size, and image resolution on the quality of explanations. Our experiments use single-stage object detection models applied to two safety-critical robotic environments: i) a shared human-robot workspace where safety is of paramount importance, and ii) an assembly area of battery kits, where safety is critical due to the potential for damage among high-risk components. Our findings evince that D-Deletion effectively gauges the performance of explanations when multiple elements of the same class appear in a scene, while D-MFPP provides a promising alternative to D-RISE when fewer masks are used.

Related papers

• On the Inherent Robustness of One-Stage Object Detection against Out-of-Distribution Data [6.267143531261792]
  We propose a novel detection algorithm for detecting unknown objects in image data. It exploits supervised dimensionality reduction techniques to mitigate the effects of the curse of dimensionality on the features extracted by the model. It utilizes high-resolution feature maps to identify potential unknown objects in an unsupervised fashion.
  arXiv  Detail & Related papers  (2024-11-07T10:15:25Z)
• Unsupervised Model Diagnosis [49.36194740479798]
  This paper proposes Unsupervised Model Diagnosis (UMO) to produce semantic counterfactual explanations without any user guidance. Our approach identifies and visualizes changes in semantics, and then matches these changes to attributes from wide-ranging text sources.
  arXiv  Detail & Related papers  (2024-10-08T17:59:03Z)
• Integrity Monitoring of 3D Object Detection in Automated Driving Systems using Raw Activation Patterns and Spatial Filtering [12.384452095533396]
  The deep neural network (DNN) models are widely used for object detection in automated driving systems (ADS) Yet, such models are prone to errors which can have serious safety implications. Introspection and self-assessment models that aim to detect such errors are therefore of paramount importance for the safe deployment of ADS.
  arXiv  Detail & Related papers  (2024-05-13T10:03:03Z)
• Open World Object Detection in the Era of Foundation Models [53.683963161370585]
  We introduce a new benchmark that includes five real-world application-driven datasets. We introduce a novel method, Foundation Object detection Model for the Open world, or FOMO, which identifies unknown objects based on their shared attributes with the base known objects.
  arXiv  Detail & Related papers  (2023-12-10T03:56:06Z)
• Innovative Horizons in Aerial Imagery: LSKNet Meets DiffusionDet for Advanced Object Detection [55.2480439325792]
  We present an in-depth evaluation of an object detection model that integrates the LSKNet backbone with the DiffusionDet head. The proposed model achieves a mean average precision (MAP) of approximately 45.7%, which is a significant improvement. This advancement underscores the effectiveness of the proposed modifications and sets a new benchmark in aerial image analysis.
  arXiv  Detail & Related papers  (2023-11-21T19:49:13Z)
• Transferring Foundation Models for Generalizable Robotic Manipulation [82.12754319808197]
  We propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models. Our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning. Demos can be found in our submitted video, and more comprehensive ones can be found in link1 or link2.
  arXiv  Detail & Related papers  (2023-06-09T07:22:12Z)
• Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability [70.72426887518517]
  Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications. We propose a novel method, Unleashing Mask, which aims to restore the OOD discriminative capabilities of the well-trained model with ID data. Our method utilizes a mask to figure out the memorized atypical samples, and then finetune the model or prune it with the introduced mask to forget them.
  arXiv  Detail & Related papers  (2023-06-06T14:23:34Z)
• Distributional Instance Segmentation: Modeling Uncertainty and High Confidence Predictions with Latent-MaskRCNN [77.0623472106488]
  In this paper, we explore a class of distributional instance segmentation models using latent codes. For robotic picking applications, we propose a confidence mask method to achieve the high precision necessary. We show that our method can significantly reduce critical errors in robotic systems, including our newly released dataset of ambiguous scenes.
  arXiv  Detail & Related papers  (2023-05-03T05:57:29Z)
• Model-agnostic explainable artificial intelligence for object detection in image data [8.042562891309414]
  Black-box explanation method named Black-box Object Detection Explanation by Masking (BODEM) We propose a hierarchical random masking framework in which coarse-grained masks are used in lower levels to find salient regions within an image. Experimentations on various object detection datasets and models showed that BODEM can effectively explain the behavior of object detectors.
  arXiv  Detail & Related papers  (2023-03-30T09:29:03Z)
• Understanding Object Detection Through An Adversarial Lens [14.976840260248913]
  This paper presents a framework for analyzing and evaluating vulnerabilities of deep object detectors under an adversarial lens. We demonstrate that the proposed framework can serve as a methodical benchmark for analyzing adversarial behaviors and risks in real-time object detection systems. We conjecture that this framework can also serve as a tool to assess the security risks and the adversarial robustness of deep object detectors to be deployed in real-world applications.
  arXiv  Detail & Related papers  (2020-07-11T18:41:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.