IMB-NAS: Neural Architecture Search for Imbalanced Datasets
- URL: http://arxiv.org/abs/2210.00136v1
- Date: Fri, 30 Sep 2022 23:15:28 GMT
- Title: IMB-NAS: Neural Architecture Search for Imbalanced Datasets
- Authors: Rahul Duggal, Shengyun Peng, Hao Zhou, Duen Horng Chau
- Abstract summary: We propose a new and complementary direction for improving performance on long tailed datasets.
We find that an architecture's accuracy obtained on a balanced dataset is not indicative of good performance on imbalanced ones.
To alleviate this compute burden, we aim to efficiently adapt a NAS super-network from a balanced source dataset to an imbalanced target one.
- Score: 18.45549536555864
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Class imbalance is a ubiquitous phenomenon occurring in real world data
distributions. To overcome its detrimental effect on training accurate
classifiers, existing work follows three major directions: class re-balancing,
information transfer, and representation learning. In this paper, we propose a
new and complementary direction for improving performance on long tailed
datasets - optimizing the backbone architecture through neural architecture
search (NAS). We find that an architecture's accuracy obtained on a balanced
dataset is not indicative of good performance on imbalanced ones. This poses
the need for a full NAS run on long tailed datasets which can quickly become
prohibitively compute intensive. To alleviate this compute burden, we aim to
efficiently adapt a NAS super-network from a balanced source dataset to an
imbalanced target one. Among several adaptation strategies, we find that the
most effective one is to retrain the linear classification head with reweighted
loss, while freezing the backbone NAS super-network trained on a balanced
source dataset. We perform extensive experiments on multiple datasets and
provide concrete insights to optimize architectures for long tailed datasets.
Related papers
- An Efficient NAS-based Approach for Handling Imbalanced Datasets [0.0]
This paper introduces a novel approach to enhance performance on long-tailed datasets by optimizing the backbone architecture through neural architecture search (NAS)
Our research shows that an architecture's accuracy on a balanced dataset does not reliably predict its performance on imbalanced datasets.
A detailed description of the fundamental techniques for IMB-NAS is provided in this paper, including NAS and architecture transfer.
arXiv Detail & Related papers (2024-06-22T15:46:03Z) - Data Augmentations in Deep Weight Spaces [89.45272760013928]
We introduce a novel augmentation scheme based on the Mixup method.
We evaluate the performance of these techniques on existing benchmarks as well as new benchmarks we generate.
arXiv Detail & Related papers (2023-11-15T10:43:13Z) - UnbiasedNets: A Dataset Diversification Framework for Robustness Bias
Alleviation in Neural Networks [11.98126285848966]
Even the most accurate NNs can be biased toward a specific output classification due to the inherent bias in the available training datasets.
This paper deals with the robustness bias, i.e., the bias exhibited by the trained NN by having a significantly large robustness to noise for a certain output class.
We propose the UnbiasedNets framework, which leverages K-means clustering and the NN's noise tolerance to diversify the given training dataset.
arXiv Detail & Related papers (2023-02-24T09:49:43Z) - Self-Supervised Neural Architecture Search for Imbalanced Datasets [129.3987858787811]
Neural Architecture Search (NAS) provides state-of-the-art results when trained on well-curated datasets with annotated labels.
We propose a NAS-based framework that bears the threefold contributions: (a) we focus on the self-supervised scenario, where no labels are required to determine the architecture, and (b) we assume the datasets are imbalanced.
arXiv Detail & Related papers (2021-09-17T14:56:36Z) - Class Balancing GAN with a Classifier in the Loop [58.29090045399214]
We introduce a novel theoretically motivated Class Balancing regularizer for training GANs.
Our regularizer makes use of the knowledge from a pre-trained classifier to ensure balanced learning of all the classes in the dataset.
We demonstrate the utility of our regularizer in learning representations for long-tailed distributions via achieving better performance than existing approaches over multiple datasets.
arXiv Detail & Related papers (2021-06-17T11:41:30Z) - AdaGCN:Adaptive Boosting Algorithm for Graph Convolutional Networks on
Imbalanced Node Classification [10.72543417177307]
We propose an ensemble model called AdaGCN, which uses a Graph Convolutional Network (GCN) as the base estimator during adaptive boosting.
Our model also improves state-of-the-art baselines on all of the challenging node classification tasks we consider.
arXiv Detail & Related papers (2021-05-25T02:43:31Z) - Task-Adaptive Neural Network Retrieval with Meta-Contrastive Learning [34.27089256930098]
We propose a novel neural network retrieval method, which retrieves the most optimal pre-trained network for a given task.
We train this framework by meta-learning a cross-modal latent space with contrastive loss, to maximize the similarity between a dataset and a network.
We validate the efficacy of our method on ten real-world datasets, against existing NAS baselines.
arXiv Detail & Related papers (2021-03-02T06:30:51Z) - Fitting the Search Space of Weight-sharing NAS with Graph Convolutional
Networks [100.14670789581811]
We train a graph convolutional network to fit the performance of sampled sub-networks.
With this strategy, we achieve a higher rank correlation coefficient in the selected set of candidates.
arXiv Detail & Related papers (2020-04-17T19:12:39Z) - Understanding the Effects of Data Parallelism and Sparsity on Neural
Network Training [126.49572353148262]
We study two factors in neural network training: data parallelism and sparsity.
Despite their promising benefits, understanding of their effects on neural network training remains elusive.
arXiv Detail & Related papers (2020-03-25T10:49:22Z) - Large-Scale Gradient-Free Deep Learning with Recursive Local
Representation Alignment [84.57874289554839]
Training deep neural networks on large-scale datasets requires significant hardware resources.
Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize.
We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
arXiv Detail & Related papers (2020-02-10T16:20:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.