DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

• URL: http://arxiv.org/abs/2303.04878v5
• Date: Thu, 29 Feb 2024 23:37:29 GMT
• Title: DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks
• Authors: Zohreh Aghababaeyan, Manel Abdellatif, Mahboubeh Dadkhah, Lionel Briand
• Abstract summary: DeepGD is a black-box multi-objective test selection approach for Deep neural networks (DNNs) It reduces the cost of labeling by prioritizing the selection of test inputs with high fault revealing power from large unlabeled datasets.
• Score: 0.6249768559720121
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. Particularly, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this paper, we propose DeepGD, a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault revealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability: (1) White-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.

Related papers

• DeepSample: DNN sampling-based testing for operational accuracy assessment [12.029919627622954]
  Deep Neural Networks (DNN) are core components for classification and regression tasks of many software systems. The challenge is to select a representative set of test inputs as small as possible to reduce the labelling cost. This study presents DeepSample, a family of DNN testing techniques for cost-effective accuracy assessment.
  arXiv  Detail & Related papers  (2024-03-28T09:56:26Z)
• GOODAT: Towards Test-time Graph Out-of-Distribution Detection [103.40396427724667]
  Graph neural networks (GNNs) have found widespread application in modeling graph data across diverse domains. Recent studies have explored graph OOD detection, often focusing on training a specific model or modifying the data on top of a well-trained GNN. This paper introduces a data-centric, unsupervised, and plug-and-play solution that operates independently of training data and modifications of GNN architecture.
  arXiv  Detail & Related papers  (2024-01-10T08:37:39Z)
• D2 Pruning: Message Passing for Balancing Diversity and Difficulty in Data Pruning [70.98091101459421]
  Coreset selection seeks to select a subset of the training data so as to maximize the performance of models trained on this subset, also referred to as coreset. We propose a novel pruning algorithm, D2 Pruning, that uses forward and reverse message passing over this dataset graph for coreset selection. Results show that D2 Pruning improves coreset selection over previous state-of-the-art methods for up to 70% pruning rates.
  arXiv  Detail & Related papers  (2023-10-11T23:01:29Z)
• Neuron Sensitivity Guided Test Case Selection for Deep Learning Testing [6.686765165569934]
  Deep Neural Networks (DNNs) have been widely deployed in software to address various tasks. DNN developers often collect rich unlabeled datasets from the natural world and label them to test the DNN models. We propose NSS, Neuron Sensitivity guided test case Selection, which can reduce the labeling time by selecting valuable test cases from unlabeled datasets.
  arXiv  Detail & Related papers  (2023-07-20T16:36:04Z)
• Debiased Graph Neural Networks with Agnostic Label Selection Bias [59.61301255860836]
  Most existing Graph Neural Networks (GNNs) are proposed without considering the selection bias in data. We propose a novel Debiased Graph Neural Networks (DGNN) with a differentiated decorrelation regularizer. Our proposed model outperforms the state-of-the-art methods and DGNN is a flexible framework to enhance existing GNNs.
  arXiv  Detail & Related papers  (2022-01-19T16:50:29Z)
• Black-Box Testing of Deep Neural Networks through Test Case Diversity [1.4700751484033807]
  We investigate black-box input diversity metrics as an alternative to white-box coverage criteria. Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria.
  arXiv  Detail & Related papers  (2021-12-20T20:12:53Z)
• Semi-supervised deep learning based on label propagation in a 2D embedded space [117.9296191012968]
  Proposed solutions propagate labels from a small set of supervised images to a large set of unsupervised ones to train a deep neural network model. We present a loop in which a deep neural network (VGG-16) is trained from a set with more correctly labeled samples along iterations. As the labeled set improves along iterations, it improves the features of the neural network.
  arXiv  Detail & Related papers  (2020-08-02T20:08:54Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)
• SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation [0.0]
  Testing of Deep Neural Networks (DNNs) has become increasingly important as DNNs are widely adopted by safety critical systems. Current testing techniques for DNNs depend on small local perturbations to existing inputs. We propose new ways to search not over the entire image space, but rather over a plausible input space that resembles the true training distribution.
  arXiv  Detail & Related papers  (2020-05-19T09:06:21Z)
• Computing the Testing Error without a Testing Set [33.068870286618655]
  We derive an algorithm to estimate the performance gap between training and testing that does not require any testing dataset. This allows us to compute the DNN's testing error on unseen samples, even when we do not have access to them.
  arXiv  Detail & Related papers  (2020-05-01T15:35:50Z)
• GraN: An Efficient Gradient-Norm Based Detector for Adversarial and Misclassified Examples [77.99182201815763]
  Deep neural networks (DNNs) are vulnerable to adversarial examples and other data perturbations. GraN is a time- and parameter-efficient method that is easily adaptable to any DNN. GraN achieves state-of-the-art performance on numerous problem set-ups.
  arXiv  Detail & Related papers  (2020-04-20T10:09:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.