Automatic Test Suite Generation for Key-points Detection DNNs Using Many-Objective Search

• URL: http://arxiv.org/abs/2012.06511v1
• Date: Fri, 11 Dec 2020 17:28:03 GMT
• Title: Automatic Test Suite Generation for Key-points Detection DNNs Using Many-Objective Search
• Authors: Fitash Ul Haq, Donghwan Shin, Lionel C. Briand, Thomas Stifter, Jun Wang
• Abstract summary: We present an approach to automatically generate test data for KP-DNNs using many-objective search. We show that our approach can generate test suites to severely mispredict, on average, more than 93% of all key-points. In comparison, random search-based test data generation can only severely mispredict 41% of them.
• Score: 12.312494463326269
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatically detecting the positions of key-points (e.g., facial key-points or finger key-points) in an image is an essential problem in many applications, such as driver's gaze detection and drowsiness detection in automated driving systems. With the recent advances of Deep Neural Networks (DNNs), Key-Points detection DNNs (KP-DNNs) have been increasingly employed for that purpose. Nevertheless, KP-DNN testing and validation have remained a challenging problem because KP-DNNs predict many independent key-points at the same time -- where each individual key-point may be critical in the targeted application -- and images can vary a great deal according to many factors. In this paper, we present an approach to automatically generate test data for KP-DNNs using many-objective search. In our experiments, focused on facial key-points detection DNNs developed for an industrial automotive application, we show that our approach can generate test suites to severely mispredict, on average, more than 93% of all key-points. In comparison, random search-based test data generation can only severely mispredict 41% of them. Many of these mispredictions, however, are not avoidable and should not therefore be considered failures. We also empirically compare state-of-the-art, many-objective search algorithms and their variants, tailored for test suite generation. Furthermore, we investigate and demonstrate how to learn specific conditions, based on image characteristics (e.g., head posture and skin color), that lead to severe mispredictions. Such conditions serve as a basis for risk analysis or DNN retraining.

Related papers

• Search-based DNN Testing and Retraining with GAN-enhanced Simulations [2.362412515574206]
  In safety-critical systems, Deep Neural Networks (DNNs) are becoming a key component for computer vision tasks. We propose to combine meta-heuristic search, used to explore the input space using simulators, with Generative Adversarial Networks (GANs) to transform the data generated by simulators into realistic input images.
  arXiv  Detail & Related papers  (2024-06-19T09:05:16Z)
• Systematic Review: Anomaly Detection in Connected and Autonomous Vehicles [0.0]
  This systematic review focuses on anomaly detection for connected and autonomous vehicles. The most commonly used Artificial Intelligence (AI) algorithms employed in anomaly detection are neural networks like LSTM, CNN, and autoencoders, alongside one-class SVM. There is a need for future research to investigate the deployment of anomaly detection to a vehicle to assess its performance on the road.
  arXiv  Detail & Related papers  (2024-05-04T18:31:38Z)
• Utilizing Radiomic Feature Analysis For Automated MRI Keypoint Detection: Enhancing Graph Applications [2.8084568003406316]
  Graph neural networks (GNNs) present a promising alternative to CNNs and transformers in certain image processing applications. One approach involves converting images into nodes by identifying significant keypoints within them. This research sets the stage for expanding GNN applications into various applications, including but not limited to image classification, segmentation, and registration.
  arXiv  Detail & Related papers  (2023-11-30T06:37:02Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• The #DNN-Verification Problem: Counting Unsafe Inputs for Deep Neural Networks [94.63547069706459]
  #DNN-Verification problem involves counting the number of input configurations of a DNN that result in a violation of a safety property. We propose a novel approach that returns the exact count of violations. We present experimental results on a set of safety-critical benchmarks.
  arXiv  Detail & Related papers  (2023-01-17T18:32:01Z)
• DiverGet: A Search-Based Software Testing Approach for Deep Neural Network Quantization Assessment [10.18462284491991]
  Quantization is one of the most applied Deep Neural Network (DNN) compression strategies. We present DiverGet, a search-based testing framework for quantization assessment. We evaluate the performance of DiverGet on state-of-the-art DNNs applied to hyperspectral remote sensing images.
  arXiv  Detail & Related papers  (2022-07-13T15:27:51Z)
• Learning to Detect Critical Nodes in Sparse Graphs via Feature Importance Awareness [53.351863569314794]
  The critical node problem (CNP) aims to find a set of critical nodes from a network whose deletion maximally degrades the pairwise connectivity of the residual network. This work proposes a feature importance-aware graph attention network for node representation. It combines it with dueling double deep Q-network to create an end-to-end algorithm to solve CNP for the first time.
  arXiv  Detail & Related papers  (2021-12-03T14:23:05Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [79.18710225716791]
  Machine learning (ML) systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a distribution different from training data. Existing OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. We propose Neural Architecture Distribution Search (NADS) to identify common building blocks among all uncertainty-aware architectures.
  arXiv  Detail & Related papers  (2020-06-11T17:39:07Z)
• A cognitive based Intrusion detection system [0.0]
  Intrusion detection is one of the important mechanisms that provide computer networks security. This paper proposes a new approach based on Deep Neural Network ans Support vector machine classifier. The proposed model predicts the attacks with better accuracy for intrusion detection rather similar methods.
  arXiv  Detail & Related papers  (2020-05-19T13:30:30Z)
• Key Points Estimation and Point Instance Segmentation Approach for Lane Detection [65.37887088194022]
  We propose a traffic line detection method called Point Instance Network (PINet) The PINet includes several stacked hourglass networks that are trained simultaneously. The PINet achieves competitive accuracy and false positive on the TuSimple and Culane datasets.
  arXiv  Detail & Related papers  (2020-02-16T15:51:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.