Input margins can predict generalization too
- URL: http://arxiv.org/abs/2308.15466v1
- Date: Tue, 29 Aug 2023 17:47:42 GMT
- Title: Input margins can predict generalization too
- Authors: Coenraad Mouton, Marthinus W. Theunissen, Marelie H. Davel
- Abstract summary: We show that input margins are not generally predictive of generalization, they can be if the search space is appropriately constrained.
We develop such a measure based on input margins, which we refer to as constrained margins'
We find that constrained margins achieve highly competitive scores and outperform other margin measurements in general.
- Score: 1.2430809884830318
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Understanding generalization in deep neural networks is an active area of
research. A promising avenue of exploration has been that of margin
measurements: the shortest distance to the decision boundary for a given sample
or its representation internal to the network. While margins have been shown to
be correlated with the generalization ability of a model when measured at its
hidden representations (hidden margins), no such link between large margins and
generalization has been established for input margins. We show that while input
margins are not generally predictive of generalization, they can be if the
search space is appropriately constrained. We develop such a measure based on
input margins, which we refer to as `constrained margins'. The predictive power
of this new measure is demonstrated on the 'Predicting Generalization in Deep
Learning' (PGDL) dataset and contrasted with hidden representation margins. We
find that constrained margins achieve highly competitive scores and outperform
other margin measurements in general. This provides a novel insight on the
relationship between generalization and classification margins, and highlights
the importance of considering the data manifold for investigations of
generalization in DNNs.
Related papers
- Editable Fairness: Fine-Grained Bias Mitigation in Language Models [52.66450426729818]
We propose a novel debiasing approach, Fairness Stamp (FAST), which enables fine-grained calibration of individual social biases.
FAST surpasses state-of-the-art baselines with superior debiasing performance.
This highlights the potential of fine-grained debiasing strategies to achieve fairness in large language models.
arXiv Detail & Related papers (2024-08-07T17:14:58Z) - On margin-based generalization prediction in deep neural networks [0.0]
We investigate margin-based generalization prediction methods in different settings.
We find that different types of sample noise can have a very different effect on the overall margin of a network.
We introduce a new margin-based measure that incorporates an approximation of the underlying data manifold.
arXiv Detail & Related papers (2024-05-20T13:26:57Z) - Marginal Debiased Network for Fair Visual Recognition [59.05212866862219]
We propose a novel marginal debiased network (MDN) to learn debiased representations.
Our MDN can achieve a remarkable performance on under-represented samples.
arXiv Detail & Related papers (2024-01-04T08:57:09Z) - The Missing Margin: How Sample Corruption Affects Distance to the
Boundary in ANNs [2.65558931169264]
We show that some types of training samples are modelled with consistently small margins while affecting generalization in different ways.
We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.
arXiv Detail & Related papers (2023-02-14T09:25:50Z) - Self-supervised debiasing using low rank regularization [59.84695042540525]
Spurious correlations can cause strong biases in deep neural networks, impairing generalization ability.
We propose a self-supervised debiasing framework potentially compatible with unlabeled samples.
Remarkably, the proposed debiasing framework significantly improves the generalization performance of self-supervised learning baselines.
arXiv Detail & Related papers (2022-10-11T08:26:19Z) - Bounding generalization error with input compression: An empirical study
with infinite-width networks [16.17600110257266]
Estimating the Generalization Error (GE) of Deep Neural Networks (DNNs) is an important task that often relies on availability of held-out data.
In search of a quantity relevant to GE, we investigate the Mutual Information (MI) between the input and final layer representations.
An existing input compression-based GE bound is used to link MI and GE.
arXiv Detail & Related papers (2022-07-19T17:05:02Z) - Bayesian Model Selection, the Marginal Likelihood, and Generalization [49.19092837058752]
We first revisit the appealing properties of the marginal likelihood for learning constraints and hypothesis testing.
We show how marginal likelihood can be negatively correlated with generalization, with implications for neural architecture search.
We also re-examine the connection between the marginal likelihood and PAC-Bayes bounds and use this connection to further elucidate the shortcomings of the marginal likelihood for model selection.
arXiv Detail & Related papers (2022-02-23T18:38:16Z) - Measuring Generalization with Optimal Transport [111.29415509046886]
We develop margin-based generalization bounds, where the margins are normalized with optimal transport costs.
Our bounds robustly predict the generalization error, given training data and network parameters, on large scale datasets.
arXiv Detail & Related papers (2021-06-07T03:04:59Z) - A Survey on Assessing the Generalization Envelope of Deep Neural
Networks: Predictive Uncertainty, Out-of-distribution and Adversarial Samples [77.99182201815763]
Deep Neural Networks (DNNs) achieve state-of-the-art performance on numerous applications.
It is difficult to tell beforehand if a DNN receiving an input will deliver the correct output since their decision criteria are usually nontransparent.
This survey connects the three fields within the larger framework of investigating the generalization performance of machine learning methods and in particular DNNs.
arXiv Detail & Related papers (2020-08-21T09:12:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.