Fantastic DNN Classifiers and How to Identify them without Data
- URL: http://arxiv.org/abs/2305.15563v1
- Date: Wed, 24 May 2023 20:54:48 GMT
- Title: Fantastic DNN Classifiers and How to Identify them without Data
- Authors: Nathaniel Dean and Dilip Sarkar
- Abstract summary: We show that the quality of a trained DNN classifier can be assessed without any example data.
We have developed two metrics: one using the features of the prototypes and the other using adversarial examples corresponding to each prototype.
Empirical evaluations show that accuracy obtained from test examples is directly proportional to quality measures obtained from the proposed metrics.
- Score: 0.685316573653194
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Current algorithms and architecture can create excellent DNN classifier
models from example data. In general, larger training datasets result in better
model estimations, which improve test performance. Existing methods for
predicting generalization performance are based on hold-out test examples. To
the best of our knowledge, at present no method exists that can estimate the
quality of a trained DNN classifier without test data. In this paper, we show
that the quality of a trained DNN classifier can be assessed without any
example data. We consider DNNs to be composed of a feature extractor and a
feature classifier; the feature extractor's output is fed to the classifier.
The proposed method iteratively creates class prototypes in the input space for
each class by minimizing a cross-entropy loss function at the output of the
network. We use these prototypes and their feature relationships to reveal the
quality of the classifier. We have developed two metrics: one using the
features of the prototypes and the other using adversarial examples
corresponding to each prototype. Empirical evaluations show that accuracy
obtained from test examples is directly proportional to quality measures
obtained from the proposed metrics. We report our observations for ResNet18
with Tiny ImageNet, CIFAR100, and CIFAR10 datasets. The proposed metrics can be
used to compare performances of two or more classifiers without test examples.
Related papers
- Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation [1.6574413179773757]
In this work, we evaluate a Deep Neural Network (DNN) classifier's training quality without any example dataset.
Our empirical study of the proposed method for ResNet18, trained with CAFIR10 and CAFIR100 datasets, confirms that data-less evaluation of DNN classifiers is indeed possible.
arXiv Detail & Related papers (2024-07-17T20:40:46Z) - FeCAM: Exploiting the Heterogeneity of Class Distributions in
Exemplar-Free Continual Learning [21.088762527081883]
Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks.
Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention.
We explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes.
arXiv Detail & Related papers (2023-09-25T11:54:33Z) - Rethinking Person Re-identification from a Projection-on-Prototypes
Perspective [84.24742313520811]
Person Re-IDentification (Re-ID) as a retrieval task, has achieved tremendous development over the past decade.
We propose a new baseline ProNet, which innovatively reserves the function of the classifier at the inference stage.
Experiments on four benchmarks demonstrate that our proposed ProNet is simple yet effective, and significantly beats previous baselines.
arXiv Detail & Related papers (2023-08-21T13:38:10Z) - Learning Prototype Classifiers for Long-Tailed Recognition [18.36167187657728]
We show that learning prototype classifiers addresses the biased softmax problem in long-tailed recognition.
We propose to jointly learn prototypes by using distances to prototypes in representation space as the logit scores for classification.
Our analysis shows that prototypes learned by Prototype classifiers are better separated than empirical centroids.
arXiv Detail & Related papers (2023-02-01T15:02:58Z) - Mutual Information Learned Classifiers: an Information-theoretic
Viewpoint of Training Deep Learning Classification Systems [9.660129425150926]
Cross entropy loss can easily lead us to find models which demonstrate severe overfitting behavior.
In this paper, we prove that the existing cross entropy loss minimization for training DNN classifiers essentially learns the conditional entropy of the underlying data distribution.
We propose a mutual information learning framework where we train DNN classifiers via learning the mutual information between the label and input.
arXiv Detail & Related papers (2022-10-03T15:09:19Z) - Combining Feature and Instance Attribution to Detect Artifacts [62.63504976810927]
We propose methods to facilitate identification of training data artifacts.
We show that this proposed training-feature attribution approach can be used to uncover artifacts in training data.
We execute a small user study to evaluate whether these methods are useful to NLP researchers in practice.
arXiv Detail & Related papers (2021-07-01T09:26:13Z) - No Fear of Heterogeneity: Classifier Calibration for Federated Learning
with Non-IID Data [78.69828864672978]
A central challenge in training classification models in the real-world federated system is learning with non-IID data.
We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model.
Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
arXiv Detail & Related papers (2021-06-09T12:02:29Z) - Rank-R FNN: A Tensor-Based Learning Model for High-Order Data
Classification [69.26747803963907]
Rank-R Feedforward Neural Network (FNN) is a tensor-based nonlinear learning model that imposes Canonical/Polyadic decomposition on its parameters.
First, it handles inputs as multilinear arrays, bypassing the need for vectorization, and can thus fully exploit the structural information along every data dimension.
We establish the universal approximation and learnability properties of Rank-R FNN, and we validate its performance on real-world hyperspectral datasets.
arXiv Detail & Related papers (2021-04-11T16:37:32Z) - ALT-MAS: A Data-Efficient Framework for Active Testing of Machine
Learning Algorithms [58.684954492439424]
We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data.
The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
arXiv Detail & Related papers (2021-04-11T12:14:04Z) - Few-Shot Named Entity Recognition: A Comprehensive Study [92.40991050806544]
We investigate three schemes to improve the model generalization ability for few-shot settings.
We perform empirical comparisons on 10 public NER datasets with various proportions of labeled data.
We create new state-of-the-art results on both few-shot and training-free settings.
arXiv Detail & Related papers (2020-12-29T23:43:16Z) - Learning from Incomplete Features by Simultaneous Training of Neural
Networks and Sparse Coding [24.3769047873156]
This paper addresses the problem of training a classifier on a dataset with incomplete features.
We assume that different subsets of features (random or structured) are available at each data instance.
A new supervised learning method is developed to train a general classifier, using only a subset of features per sample.
arXiv Detail & Related papers (2020-11-28T02:20:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.