Machine Learning Data Suitability and Performance Testing Using Fault Injection Testing Framework

• URL: http://arxiv.org/abs/2309.11274v1
• Date: Wed, 20 Sep 2023 12:58:35 GMT
• Title: Machine Learning Data Suitability and Performance Testing Using Fault Injection Testing Framework
• Authors: Manal Rahal and Bestoun S. Ahmed and Jorgen Samuelsson
• Abstract summary: This paper presents the Fault Injection for Undesirable Learning in input Data (FIUL-Data) testing framework. Data mutators explore vulnerabilities of ML systems against the effects of different fault injections. This paper evaluates the framework using data from analytical chemistry, comprising retention time measurements of anti-sense oligonucleotides.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Creating resilient machine learning (ML) systems has become necessary to ensure production-ready ML systems that acquire user confidence seamlessly. The quality of the input data and the model highly influence the successful end-to-end testing in data-sensitive systems. However, the testing approaches of input data are not as systematic and are few compared to model testing. To address this gap, this paper presents the Fault Injection for Undesirable Learning in input Data (FIUL-Data) testing framework that tests the resilience of ML models to multiple intentionally-triggered data faults. Data mutators explore vulnerabilities of ML systems against the effects of different fault injections. The proposed framework is designed based on three main ideas: The mutators are not random; one data mutator is applied at an instance of time, and the selected ML models are optimized beforehand. This paper evaluates the FIUL-Data framework using data from analytical chemistry, comprising retention time measurements of anti-sense oligonucleotide. Empirical evaluation is carried out in a two-step process in which the responses of selected ML models to data mutation are analyzed individually and then compared with each other. The results show that the FIUL-Data framework allows the evaluation of the resilience of ML models. In most experiments cases, ML models show higher resilience at larger training datasets, where gradient boost performed better than support vector regression in smaller training sets. Overall, the mean squared error metric is useful in evaluating the resilience of models due to its higher sensitivity to data mutation.

Related papers

• Context-Aware Testing: A New Paradigm for Model Testing with Large Language Models [49.06068319380296]
  We introduce context-aware testing (CAT) which uses context as an inductive bias to guide the search for meaningful model failures. We instantiate the first CAT system, SMART Testing, which employs large language models to hypothesize relevant and likely failures.
  arXiv  Detail & Related papers  (2024-10-31T15:06:16Z)
• Retrosynthesis prediction enhanced by in-silico reaction data augmentation [66.5643280109899]
  We present RetroWISE, a framework that employs a base model inferred from real paired data to perform in-silico reaction generation and augmentation. On three benchmark datasets, RetroWISE achieves the best overall performance against state-of-the-art models.
  arXiv  Detail & Related papers  (2024-01-31T07:40:37Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• Information FOMO: The unhealthy fear of missing out on information. A method for removing misleading data for healthier models [0.0]
  Misleading or unnecessary data can have out-sized impacts on the health or accuracy of Machine Learning (ML) models. We present a sequential selection method that identifies critically important information within a dataset. We find these instabilities are a result of the complexity of the underlying map and linked to extreme events and heavy tails.
  arXiv  Detail & Related papers  (2022-08-27T19:43:53Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• Effect of Balancing Data Using Synthetic Data on the Performance of Machine Learning Classifiers for Intrusion Detection in Computer Networks [3.233545237942899]
  Researchers in academia and industry used machine learning (ML) techniques to design and implement Intrusion Detection Systems (IDSes) for computer networks. In many of the datasets used in such systems, data are imbalanced (i.e., not all classes have equal amount of samples) We show that training ML models on dataset balanced with synthetic samples generated by CTGAN increased prediction accuracy by up to $8%$.
  arXiv  Detail & Related papers  (2022-04-01T00:25:11Z)
• Classifier Data Quality: A Geometric Complexity Based Method for Automated Baseline And Insights Generation [4.722075132982135]
  A major challenge is to determine when the level of incorrectness, e.g., model accuracy or F1 score for classifiers, is acceptable. We have developed complexity measures, which quantify how difficult given observations are to assign to their true class label. These measures are superior to the best practice baseline in that, for a linear computation cost, they also quantify each observation' classification complexity in an explainable form.
  arXiv  Detail & Related papers  (2021-12-22T12:17:08Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Out-Of-Bag Anomaly Detection [0.9449650062296822]
  Data anomalies are ubiquitous in real world datasets, and can have an adverse impact on machine learning (ML) systems. We propose a novel model-based anomaly detection method, that we call Out-of-Bag anomaly detection. We show our method can improve the accuracy and reliability of an ML system as data pre-processing step via a case study on home valuation.
  arXiv  Detail & Related papers  (2020-09-20T06:01:52Z)
• Data and Model Dependencies of Membership Inference Attack [13.951470844348899]
  We provide an empirical analysis of the impact of both the data and ML model properties on the vulnerability of ML techniques to MIA. Our results reveal the relationship between MIA accuracy and properties of the dataset and training model in use. We propose using those data and model properties as regularizers to protect ML models against MIA.
  arXiv  Detail & Related papers  (2020-02-17T09:35:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.