Latent Space Class Dispersion: Effective Test Data Quality Assessment for DNNs

• URL: http://arxiv.org/abs/2503.18799v1
• Date: Mon, 24 Mar 2025 15:45:50 GMT
• Title: Latent Space Class Dispersion: Effective Test Data Quality Assessment for DNNs
• Authors: Vivek Vekariya, Mojdeh Golagha, Andrea Stocco, Alexander Pretschner,
• Abstract summary: Latent Space Class Dispersion (LSCD) is a novel metric to quantify the quality of test datasets for Deep Neural Networks (DNNs)<n>Our empirical study shows that LSCD reveals and quantifies deficiencies in the test dataset of three popular benchmarks pertaining to image classification tasks.
• Score: 45.129846925131055
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: High-quality test datasets are crucial for assessing the reliability of Deep Neural Networks (DNNs). Mutation testing evaluates test dataset quality based on their ability to uncover injected faults in DNNs as measured by mutation score (MS). At the same time, its high computational cost motivates researchers to seek alternative test adequacy criteria. We propose Latent Space Class Dispersion (LSCD), a novel metric to quantify the quality of test datasets for DNNs. It measures the degree of dispersion within a test dataset as observed in the latent space of a DNN. Our empirical study shows that LSCD reveals and quantifies deficiencies in the test dataset of three popular benchmarks pertaining to image classification tasks using DNNs. Corner cases generated using automated fuzzing were found to help enhance fault detection and improve the overall quality of the original test sets calculated by MS and LSCD. Our experiments revealed a high positive correlation (0.87) between LSCD and MS, significantly higher than the one achieved by the well-studied Distance-based Surprise Coverage (0.25). These results were obtained from 129 mutants generated through pre-training mutation operators, with statistical significance and a high validity of corner cases. These observations suggest that LSCD can serve as a cost-effective alternative to expensive mutation testing, eliminating the need to generate mutant models while offering comparably valuable insights into test dataset quality for DNNs.

Related papers

• DNN-GDITD: Out-of-distribution detection via Deep Neural Network based Gaussian Descriptor for Imbalanced Tabular Data [3.3842496750884457]
  This study introduces a novel OOD detection algorithm, titled Deep Neural Network-based Gaussian Descriptor for Imbalanced Tabular Data (DNN-GDITD) The algorithm can be placed on top of any DNN to facilitate better classification of imbalanced data and OOD detection using spherical decision boundaries.
  arXiv  Detail & Related papers  (2024-09-02T06:52:01Z)
• TEASMA: A Practical Methodology for Test Adequacy Assessment of Deep Neural Networks [4.528286105252983]
  TEASMA is a comprehensive and practical methodology designed to accurately assess the adequacy of test sets for Deep Neural Networks. We evaluate TEASMA with four state-of-the-art test adequacy metrics: Distance-based Surprise Coverage (DSC), Likelihood-based Surprise Coverage (LSC), Input Distribution Coverage (IDC) and Mutation Score (MS)
  arXiv  Detail & Related papers  (2023-08-02T17:56:05Z)
• D-Score: A White-Box Diagnosis Score for CNNs Based on Mutation Operators [8.977819892091]
  Convolutional neural networks (CNNs) have been widely applied in many safety-critical domains, such as autonomous driving and medical diagnosis. We propose a white-box diagnostic approach that uses mutation operators and image transformation to calculate the feature and attention distribution of the model. We also propose a D-Score based data augmentation method to enhance the CNN's performance to translations and rescalings.
  arXiv  Detail & Related papers  (2023-04-03T03:13:59Z)
• Energy-based Out-of-Distribution Detection for Graph Neural Networks [76.0242218180483]
  We propose a simple, powerful and efficient OOD detection model for GNN-based learning on graphs, which we call GNNSafe. GNNSafe achieves up to $17.0%$ AUROC improvement over state-of-the-arts and it could serve as simple yet strong baselines in such an under-developed area.
  arXiv  Detail & Related papers  (2023-02-06T16:38:43Z)
• Labeling-Free Comparison Testing of Deep Learning Models [28.47632100019289]
  We propose a labeling-free comparison testing approach to overcome the limitations of labeling effort and sampling randomness. Our approach outperforms the baseline methods by up to 0.74 and 0.53 on Spearman's correlation and Kendall's $tau$, regardless of the dataset and distribution shift.
  arXiv  Detail & Related papers  (2022-04-08T10:55:45Z)
• Statistical model-based evaluation of neural networks [74.10854783437351]
  We develop an experimental setup for the evaluation of neural networks (NNs) The setup helps to benchmark a set of NNs vis-a-vis minimum-mean-square-error (MMSE) performance bounds. This allows us to test the effects of training data size, data dimension, data geometry, noise, and mismatch between training and testing conditions.
  arXiv  Detail & Related papers  (2020-11-18T00:33:24Z)
• CovidDeep: SARS-CoV-2/COVID-19 Test Based on Wearable Medical Sensors and Efficient Neural Networks [51.589769497681175]
  The novel coronavirus (SARS-CoV-2) has led to a pandemic. The current testing regime based on Reverse Transcription-Polymerase Chain Reaction for SARS-CoV-2 has been unable to keep up with testing demands. We propose a framework called CovidDeep that combines efficient DNNs with commercially available WMSs for pervasive testing of the virus.
  arXiv  Detail & Related papers  (2020-07-20T21:47:28Z)
• NADS: Neural Architecture Distribution Search for Uncertainty Awareness [79.18710225716791]
  Machine learning (ML) systems often encounter Out-of-Distribution (OoD) errors when dealing with testing data coming from a distribution different from training data. Existing OoD detection approaches are prone to errors and even sometimes assign higher likelihoods to OoD samples. We propose Neural Architecture Distribution Search (NADS) to identify common building blocks among all uncertainty-aware architectures.
  arXiv  Detail & Related papers  (2020-06-11T17:39:07Z)
• Noisy Adaptive Group Testing using Bayesian Sequential Experimental Design [63.48989885374238]
  When the infection prevalence of a disease is low, Dorfman showed 80 years ago that testing groups of people can prove more efficient than testing people individually. Our goal in this paper is to propose new group testing algorithms that can operate in a noisy setting.
  arXiv  Detail & Related papers  (2020-04-26T23:41:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.