Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing

• URL: http://arxiv.org/abs/2110.00330v1
• Date: Fri, 1 Oct 2021 11:47:56 GMT
• Title: Discovering Boundary Values of Feature-based Machine Learning Classifiers through Exploratory Datamorphic Testing
• Authors: Hong Zhu and Ian Bayley
• Abstract summary: This paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. Three variants of exploratory strategies are presented with the algorithms implemented in the automated datamorphic testing tool Morphy. Their capability and cost of discovering borders between classes are evaluated via a set of controlled experiments with manually designed subjects and a set of case studies with real machine learning models.
• Score: 7.8729820663730035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Testing has been widely recognised as difficult for AI applications. This paper proposes a set of testing strategies for testing machine learning applications in the framework of the datamorphism testing methodology. In these strategies, testing aims at exploring the data space of a classification or clustering application to discover the boundaries between classes that the machine learning application defines. This enables the tester to understand precisely the behaviour and function of the software under test. In the paper, three variants of exploratory strategies are presented with the algorithms implemented in the automated datamorphic testing tool Morphy. The correctness of these algorithms are formally proved. Their capability and cost of discovering borders between classes are evaluated via a set of controlled experiments with manually designed subjects and a set of case studies with real machine learning models.

Related papers

• Survey of Computerized Adaptive Testing: A Machine Learning Perspective [66.26687542572974]
  Computerized Adaptive Testing (CAT) provides an efficient and tailored method for assessing the proficiency of examinees. This paper aims to provide a machine learning-focused survey on CAT, presenting a fresh perspective on this adaptive testing method.
  arXiv  Detail & Related papers  (2024-03-31T15:09:47Z)
• Constrained Adversarial Learning and its applicability to Automated Software Testing: a systematic review [0.0]
  This systematic review is focused on the current state-of-the-art of constrained data generation methods applied for adversarial learning and software testing. It aims to guide researchers and developers to enhance testing tools with adversarial learning methods and improve the resilience and robustness of their digital systems.
  arXiv  Detail & Related papers  (2023-03-14T00:27:33Z)
• ALBench: A Framework for Evaluating Active Learning in Object Detection [102.81795062493536]
  This paper contributes an active learning benchmark framework named as ALBench for evaluating active learning in object detection. Developed on an automatic deep model training system, this ALBench framework is easy-to-use, compatible with different active learning algorithms, and ensures the same training and testing protocols.
  arXiv  Detail & Related papers  (2022-07-27T07:46:23Z)
• Differential testing for machine learning: an analysis for classification algorithms beyond deep learning [7.081604594416339]
  We conduct a case study using Scikit-learn, Weka, Spark MLlib, and Caret. We identify the potential of differential testing by considering which algorithms are available in multiple frameworks. The feasibility seems limited because often it is not possible to determine configurations that are the same in other frameworks.
  arXiv  Detail & Related papers  (2022-07-25T08:27:01Z)
• The Integration of Machine Learning into Automated Test Generation: A Systematic Mapping Study [15.016047591601094]
  We characterize emerging research, examining testing practices, researcher goals, ML techniques applied, evaluation, and challenges. ML generates input for system, GUI, unit, performance, and testing or improves the performance of existing generation methods.
  arXiv  Detail & Related papers  (2022-06-21T09:26:25Z)
• The Familiarity Hypothesis: Explaining the Behavior of Deep Open Set Methods [86.39044549664189]
  Anomaly detection algorithms for feature-vector data identify anomalies as outliers, but outlier detection has not worked well in deep learning. This paper proposes the Familiarity Hypothesis that these methods succeed because they are detecting the absence of familiar learned features rather than the presence of novelty. The paper concludes with a discussion of whether familiarity detection is an inevitable consequence of representation learning.
  arXiv  Detail & Related papers  (2022-03-04T18:32:58Z)
• Human-in-the-Loop Disinformation Detection: Stance, Sentiment, or Something Else? [93.91375268580806]
  Both politics and pandemics have recently provided ample motivation for the development of machine learning-enabled disinformation (a.k.a. fake news) detection algorithms. Existing literature has focused primarily on the fully-automated case, but the resulting techniques cannot reliably detect disinformation on the varied topics, sources, and time scales required for military applications. By leveraging an already-available analyst as a human-in-the-loop, canonical machine learning techniques of sentiment analysis, aspect-based sentiment analysis, and stance detection become plausible methods to use for a partially-automated disinformation detection system.
  arXiv  Detail & Related papers  (2021-11-09T13:30:34Z)
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control [67.52000805944924]
  Learn then Test (LTT) is a framework for calibrating machine learning models. Our main insight is to reframe the risk-control problem as multiple hypothesis testing. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.
  arXiv  Detail & Related papers  (2021-10-03T17:42:03Z)
• Class Introspection: A Novel Technique for Detecting Unlabeled Subclasses by Leveraging Classifier Explainability Methods [0.0]
  latent structure is a crucial step in performing analysis of a dataset. By leveraging instance explanation methods, an existing classifier can be extended to detect latent classes. This paper also contains a pipeline for analyzing classifiers automatically, and a web application for interactively exploring the results from this technique.
  arXiv  Detail & Related papers  (2021-07-04T14:58:29Z)
• Hybrid Method Based on NARX models and Machine Learning for Pattern Recognition [0.0]
  This work presents a novel technique that integrates the methodologies of machine learning and system identification to solve multiclass problems. The efficiency of the method was tested by running case studies investigated in machine learning, obtaining better absolute results when compared with classical classification algorithms.
  arXiv  Detail & Related papers  (2021-06-08T00:17:36Z)
• AutoOD: Automated Outlier Detection via Curiosity-guided Search and Self-imitation Learning [72.99415402575886]
  Outlier detection is an important data mining task with numerous practical applications. We propose AutoOD, an automated outlier detection framework, which aims to search for an optimal neural network model. Experimental results on various real-world benchmark datasets demonstrate that the deep model identified by AutoOD achieves the best performance.
  arXiv  Detail & Related papers  (2020-06-19T18:57:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.