D-Score: A White-Box Diagnosis Score for CNNs Based on Mutation Operators

• URL: http://arxiv.org/abs/2304.00697v1
• Date: Mon, 3 Apr 2023 03:13:59 GMT
• Title: D-Score: A White-Box Diagnosis Score for CNNs Based on Mutation Operators
• Authors: Xin Zhang and Yuqi Song and Xiaofeng Wang and Fei Zuo
• Abstract summary: Convolutional neural networks (CNNs) have been widely applied in many safety-critical domains, such as autonomous driving and medical diagnosis. We propose a white-box diagnostic approach that uses mutation operators and image transformation to calculate the feature and attention distribution of the model. We also propose a D-Score based data augmentation method to enhance the CNN's performance to translations and rescalings.
• Score: 8.977819892091
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Convolutional neural networks (CNNs) have been widely applied in many safety-critical domains, such as autonomous driving and medical diagnosis. However, concerns have been raised with respect to the trustworthiness of these models: The standard testing method evaluates the performance of a model on a test set, while low-quality and insufficient test sets can lead to unreliable evaluation results, which can have unforeseeable consequences. Therefore, how to comprehensively evaluate CNNs and, based on the evaluation results, how to enhance their trustworthiness are the key problems to be urgently addressed. Prior work has used mutation tests to evaluate the test sets of CNNs. However, the evaluation scores are black boxes and not explicit enough for what is being tested. In this paper, we propose a white-box diagnostic approach that uses mutation operators and image transformation to calculate the feature and attention distribution of the model and further present a diagnosis score, namely D-Score, to reflect the model's robustness and fitness to a dataset. We also propose a D-Score based data augmentation method to enhance the CNN's performance to translations and rescalings. Comprehensive experiments on two widely used datasets and three commonly adopted CNNs demonstrate the effectiveness of our approach.

Related papers

• TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural Networks [4.528286105252983]
  TEASMA is a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for Deep Neural Networks. We evaluate TEASMA with four state-of-the-art test adequacy metrics: Distance-based Surprise Coverage (DSC), Likelihood-based Surprise Coverage (LSC), Input Distribution Coverage (IDC) and Mutation Score (MS)
  arXiv  Detail & Related papers  (2023-08-02T17:56:05Z)
• Towards Reliable Medical Image Segmentation by utilizing Evidential Calibrated Uncertainty [52.03490691733464]
  We introduce DEviS, an easily implementable foundational model that seamlessly integrates into various medical image segmentation networks. By leveraging subjective logic theory, we explicitly model probability and uncertainty for the problem of medical image segmentation. DeviS incorporates an uncertainty-aware filtering module, which utilizes the metric of uncertainty-calibrated error to filter reliable data.
  arXiv  Detail & Related papers  (2023-01-01T05:02:46Z)
• Stain Normalized Breast Histopathology Image Recognition using Convolutional Neural Networks for Cancer Detection [9.826027427965354]
  Recent advances have shown that the convolutional Neural Network (CNN) architectures can be used to design a Computer Aided Diagnostic (CAD) System for breast cancer detection. We consider some contemporary CNN models for binary classification of breast histopathology images. We have validated the trained CNN networks on a publicly available BreaKHis dataset, for 200x and 400x magnified histopathology images.
  arXiv  Detail & Related papers  (2022-01-04T03:09:40Z)
• Novelty-based Generalization Evaluation for Traffic Light Detection [13.487711023133764]
  We evaluate the generalization ability of Convolutional Neural Networks (CNNs) by calculating various metrics on an independent test dataset. We propose a CNN generalization scoring framework that considers novelty of objects in the test dataset.
  arXiv  Detail & Related papers  (2022-01-03T09:23:56Z)
• Black-Box Testing of Deep Neural Networks through Test Case Diversity [1.4700751484033807]
  We investigate black-box input diversity metrics as an alternative to white-box coverage criteria. Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria.
  arXiv  Detail & Related papers  (2021-12-20T20:12:53Z)
• Model Doctor: A Simple Gradient Aggregation Strategy for Diagnosing and Treating CNN Classifiers [33.82339346293966]
  Convolutional Neural Network (CNN) has achieved excellent performance in the classification task. It is widely known that CNN is deemed as a 'black-box', which is hard for understanding the prediction mechanism. We propose the first completely automatic model diagnosing and treating tool, termed as Model Doctor.
  arXiv  Detail & Related papers  (2021-12-09T14:05:00Z)
• BreakingBED -- Breaking Binary and Efficient Deep Neural Networks by Adversarial Attacks [65.2021953284622]
  We study robustness of CNNs against white-box and black-box adversarial attacks. Results are shown for distilled CNNs, agent-based state-of-the-art pruned models, and binarized neural networks.
  arXiv  Detail & Related papers  (2021-03-14T20:43:19Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• An Uncertainty-Driven GCN Refinement Strategy for Organ Segmentation [53.425900196763756]
  We propose a segmentation refinement method based on uncertainty analysis and graph convolutional networks. We employ the uncertainty levels of the convolutional network in a particular input volume to formulate a semi-supervised graph learning problem. We show that our method outperforms the state-of-the-art CRF refinement method by improving the dice score by 1% for the pancreas and 2% for spleen.
  arXiv  Detail & Related papers  (2020-12-06T18:55:07Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.