Experimenting with an Evaluation Framework for Imbalanced Data Learning
(EFIDL)
- URL: http://arxiv.org/abs/2301.10888v1
- Date: Thu, 26 Jan 2023 01:16:02 GMT
- Title: Experimenting with an Evaluation Framework for Imbalanced Data Learning
(EFIDL)
- Authors: Chenyu Li, Xia Jiang
- Abstract summary: Data imbalance is one of the crucial issues in big data analysis with fewer labels.
Many data balance methods were introduced to improve machine learning algorithms' performance.
We proposed, a new evaluation framework for imbalanced data learning methods.
- Score: 9.010643838773477
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Introduction Data imbalance is one of the crucial issues in big data analysis
with fewer labels. For example, in real-world healthcare data, spam detection
labels, and financial fraud detection datasets. Many data balance methods were
introduced to improve machine learning algorithms' performance. Research claims
SMOTE and SMOTE-based data-augmentation (generate new data points) methods
could improve algorithm performance. However, we found in many online
tutorials, the valuation methods were applied based on synthesized datasets
that introduced bias into the evaluation, and the performance got a false
improvement. In this study, we proposed, a new evaluation framework for
imbalanced data learning methods. We have experimented on five data balance
methods and whether the performance of algorithms will improve or not. Methods
We collected 8 imbalanced healthcare datasets with different imbalanced rates
from different domains. Applied 6 data augmentation methods with 11 machine
learning methods testing if the data augmentation will help with improving
machine learning performance. We compared the traditional data augmentation
evaluation methods with our proposed cross-validation evaluation framework
Results Using traditional data augmentation evaluation meta hods will give a
false impression of improving the performance. However, our proposed evaluation
method shows data augmentation has limited ability to improve the results.
Conclusion EFIDL is more suitable for evaluating the prediction performance of
an ML method when data are augmented. Using an unsuitable evaluation framework
will give false results. Future researchers should consider the evaluation
framework we proposed when dealing with augmented datasets. Our experiments
showed data augmentation does not help improve ML prediction performance.
Related papers
- What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions [34.99034454081842]
Large language models (LLMs) are trained on a vast amount of human-written data, but data providers often remain uncredited.
In this work, we focus on influence functions, a popular gradient-based data valuation method, and significantly improve its scalability.
We also introduce LogIX, a software package that can transform existing training code into data valuation code with minimal effort.
arXiv Detail & Related papers (2024-05-22T19:39:05Z) - Neural Dynamic Data Valuation [4.286118155737111]
We propose a novel data valuation method from the perspective of optimal control, named the neural dynamic data valuation (NDDV)
Our method has solid theoretical interpretations to accurately identify the data valuation via the sensitivity of the data optimal control state.
In addition, we implement a data re-weighting strategy to capture the unique features of data points, ensuring fairness through the interaction between data points and the mean-field states.
arXiv Detail & Related papers (2024-04-30T13:39:26Z) - LAVA: Data Valuation without Pre-Specified Learning Algorithms [20.578106028270607]
We introduce a new framework that can value training data in a way that is oblivious to the downstream learning algorithm.
We develop a proxy for the validation performance associated with a training set based on a non-conventional class-wise Wasserstein distance between training and validation sets.
We show that the distance characterizes the upper bound of the validation performance for any given model under certain Lipschitz conditions.
arXiv Detail & Related papers (2023-04-28T19:05:16Z) - A review of ensemble learning and data augmentation models for class
imbalanced problems: combination, implementation and evaluation [0.196629787330046]
Class imbalance (CI) in classification problems arises when the number of observations belonging to one class is lower than the other.
In this paper, we evaluate data augmentation and ensemble learning methods used to address prominent benchmark CI problems.
arXiv Detail & Related papers (2023-04-06T04:37:10Z) - Revisiting Long-tailed Image Classification: Survey and Benchmarks with
New Evaluation Metrics [88.39382177059747]
A corpus of metrics is designed for measuring the accuracy, robustness, and bounds of algorithms for learning with long-tailed distribution.
Based on our benchmarks, we re-evaluate the performance of existing methods on CIFAR10 and CIFAR100 datasets.
arXiv Detail & Related papers (2023-02-03T02:40:54Z) - Augmentation-Aware Self-Supervision for Data-Efficient GAN Training [68.81471633374393]
Training generative adversarial networks (GANs) with limited data is challenging because the discriminator is prone to overfitting.
We propose a novel augmentation-aware self-supervised discriminator that predicts the augmentation parameter of the augmented data.
We compare our method with state-of-the-art (SOTA) methods using the class-conditional BigGAN and unconditional StyleGAN2 architectures.
arXiv Detail & Related papers (2022-05-31T10:35:55Z) - Invariance Learning in Deep Neural Networks with Differentiable Laplace
Approximations [76.82124752950148]
We develop a convenient gradient-based method for selecting the data augmentation.
We use a differentiable Kronecker-factored Laplace approximation to the marginal likelihood as our objective.
arXiv Detail & Related papers (2022-02-22T02:51:11Z) - Data-Centric Machine Learning in the Legal Domain [0.2624902795082451]
This paper explores how changes in a data set influence the measured performance of a model.
Using three publicly available data sets from the legal domain, we investigate how changes to their size, the train/test splits, and the human labelling accuracy impact the performance.
The observed effects are surprisingly pronounced, especially when the per-class performance is considered.
arXiv Detail & Related papers (2022-01-17T23:05:14Z) - Doing Great at Estimating CATE? On the Neglected Assumptions in
Benchmark Comparisons of Treatment Effect Estimators [91.3755431537592]
We show that even in arguably the simplest setting, estimation under ignorability assumptions can be misleading.
We consider two popular machine learning benchmark datasets for evaluation of heterogeneous treatment effect estimators.
We highlight that the inherent characteristics of the benchmark datasets favor some algorithms over others.
arXiv Detail & Related papers (2021-07-28T13:21:27Z) - Can Active Learning Preemptively Mitigate Fairness Issues? [66.84854430781097]
dataset bias is one of the prevailing causes of unfairness in machine learning.
We study whether models trained with uncertainty-based ALs are fairer in their decisions with respect to a protected class.
We also explore the interaction of algorithmic fairness methods such as gradient reversal (GRAD) and BALD.
arXiv Detail & Related papers (2021-04-14T14:20:22Z) - Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training.
We experimentally verify that the new dataset can significantly improve the ability of the learned FER model.
To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
arXiv Detail & Related papers (2020-05-18T09:36:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.