Novelty-based Generalization Evaluation for Traffic Light Detection
- URL: http://arxiv.org/abs/2201.00531v1
- Date: Mon, 3 Jan 2022 09:23:56 GMT
- Title: Novelty-based Generalization Evaluation for Traffic Light Detection
- Authors: Arvind Kumar Shekar, Laureen Lake, Liang Gou, Liu Ren
- Abstract summary: We evaluate the generalization ability of Convolutional Neural Networks (CNNs) by calculating various metrics on an independent test dataset.
We propose a CNN generalization scoring framework that considers novelty of objects in the test dataset.
- Score: 13.487711023133764
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The advent of Convolutional Neural Networks (CNNs) has led to their
application in several domains. One noteworthy application is the perception
system for autonomous driving that relies on the predictions from CNNs.
Practitioners evaluate the generalization ability of such CNNs by calculating
various metrics on an independent test dataset. A test dataset is often chosen
based on only one precondition, i.e., its elements are not a part of the
training data. Such a dataset may contain objects that are both similar and
novel w.r.t. the training dataset. Nevertheless, existing works do not reckon
the novelty of the test samples and treat them all equally for evaluating
generalization. Such novelty-based evaluations are of significance to validate
the fitness of a CNN in autonomous driving applications. Hence, we propose a
CNN generalization scoring framework that considers novelty of objects in the
test dataset. We begin with the representation learning technique to reduce the
image data into a low-dimensional space. It is on this space we estimate the
novelty of the test samples. Finally, we calculate the generalization score as
a combination of the test data prediction performance and novelty. We perform
an experimental study of the same for our traffic light detection application.
In addition, we systematically visualize the results for an interpretable
notion of novelty.
Related papers
- Downstream-Pretext Domain Knowledge Traceback for Active Learning [138.02530777915362]
We propose a downstream-pretext domain knowledge traceback (DOKT) method that traces the data interactions of downstream knowledge and pre-training guidance.
DOKT consists of a traceback diversity indicator and a domain-based uncertainty estimator.
Experiments conducted on ten datasets show that our model outperforms other state-of-the-art methods.
arXiv Detail & Related papers (2024-07-20T01:34:13Z) - Novel Deep Neural Network Classifier Characterization Metrics with Applications to Dataless Evaluation [1.6574413179773757]
In this work, we evaluate a Deep Neural Network (DNN) classifier's training quality without any example dataset.
Our empirical study of the proposed method for ResNet18, trained with CAFIR10 and CAFIR100 datasets, confirms that data-less evaluation of DNN classifiers is indeed possible.
arXiv Detail & Related papers (2024-07-17T20:40:46Z) - When do Convolutional Neural Networks Stop Learning? [0.0]
Convolutional Neural Networks (CNNs) have demonstrated outstanding performance in computer vision tasks.
Current practice is to stop training when the training loss decreases and the gap between training and validation error increases.
This research work introduces a hypothesis that analyzes the data variation across all the layers of a CNN variant to anticipate its near-optimal learning capacity.
arXiv Detail & Related papers (2024-03-04T20:35:09Z) - Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls
and New Benchmarking [66.83273589348758]
Link prediction attempts to predict whether an unseen edge exists based on only a portion of edges of a graph.
A flurry of methods have been introduced in recent years that attempt to make use of graph neural networks (GNNs) for this task.
New and diverse datasets have also been created to better evaluate the effectiveness of these new models.
arXiv Detail & Related papers (2023-06-18T01:58:59Z) - Conformal prediction for the design problem [72.14982816083297]
In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next.
In such settings, there is a distinct type of distribution shift between the training and test data.
We introduce a method to quantify predictive uncertainty in such settings.
arXiv Detail & Related papers (2022-02-08T02:59:12Z) - Towards Open-World Feature Extrapolation: An Inductive Graph Learning
Approach [80.8446673089281]
We propose a new learning paradigm with graph representation and learning.
Our framework contains two modules: 1) a backbone network (e.g., feedforward neural nets) as a lower model takes features as input and outputs predicted labels; 2) a graph neural network as an upper model learns to extrapolate embeddings for new features via message passing over a feature-data graph built from observed data.
arXiv Detail & Related papers (2021-10-09T09:02:45Z) - Dataset Bias Mitigation Through Analysis of CNN Training Scores [0.0]
We propose a novel, domain-independent approach, called score-based resampling (SBR), to locate the under-represented samples of the original training dataset.
In our method, once trained, we use the same CNN model to infer on its own training samples, obtain prediction scores, and based on the distance between predicted and ground-truth, we identify samples that are far away from their ground-truth.
The obtained results confirmed the validity of our proposed method regrading identifying under-represented samples among original dataset to decrease categorical bias of classifying certain groups.
arXiv Detail & Related papers (2021-06-28T16:07:49Z) - ALT-MAS: A Data-Efficient Framework for Active Testing of Machine
Learning Algorithms [58.684954492439424]
We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data.
The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
arXiv Detail & Related papers (2021-04-11T12:14:04Z) - Learning from Incomplete Features by Simultaneous Training of Neural
Networks and Sparse Coding [24.3769047873156]
This paper addresses the problem of training a classifier on a dataset with incomplete features.
We assume that different subsets of features (random or structured) are available at each data instance.
A new supervised learning method is developed to train a general classifier, using only a subset of features per sample.
arXiv Detail & Related papers (2020-11-28T02:20:39Z) - Statistical model-based evaluation of neural networks [74.10854783437351]
We develop an experimental setup for the evaluation of neural networks (NNs)
The setup helps to benchmark a set of NNs vis-a-vis minimum-mean-square-error (MMSE) performance bounds.
This allows us to test the effects of training data size, data dimension, data geometry, noise, and mismatch between training and testing conditions.
arXiv Detail & Related papers (2020-11-18T00:33:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.