The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs

• URL: http://arxiv.org/abs/2302.06925v1
• Date: Tue, 14 Feb 2023 09:25:50 GMT
• Title: The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs
• Authors: Marthinus W. Theunissen and Coenraad Mouton and Marelie H. Davel
• Abstract summary: We show that some types of training samples are modelled with consistently small margins while affecting generalization in different ways. We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.
• Score: 2.65558931169264
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Classification margins are commonly used to estimate the generalization ability of machine learning models. We present an empirical study of these margins in artificial neural networks. A global estimate of margin size is usually used in the literature. In this work, we point out seldom considered nuances regarding classification margins. Notably, we demonstrate that some types of training samples are modelled with consistently small margins while affecting generalization in different ways. By showing a link with the minimum distance to a different-target sample and the remoteness of samples from one another, we provide a plausible explanation for this observation. We support our findings with an analysis of fully-connected networks trained on noise-corrupted MNIST data, as well as convolutional networks trained on noise-corrupted CIFAR10 data.

Related papers

• An accurate detection is not all you need to combat label noise in web-noisy datasets [23.020126612431746]
  We show that direct estimation of the separating hyperplane can indeed offer an accurate detection of OOD samples. We propose a hybrid solution that alternates between noise detection using linear separation and a state-of-the-art (SOTA) small-loss approach.
  arXiv  Detail & Related papers  (2024-07-08T00:21:42Z)
• On margin-based generalization prediction in deep neural networks [0.0]
  We investigate margin-based generalization prediction methods in different settings. We find that different types of sample noise can have a very different effect on the overall margin of a network. We introduce a new margin-based measure that incorporates an approximation of the underlying data manifold.
  arXiv  Detail & Related papers  (2024-05-20T13:26:57Z)
• Theoretical Understanding of Learning from Adversarial Perturbations [30.759348459463467]
  It is not fully understood why adversarial examples can deceive neural networks and transfer between different networks. We provide a theoretical framework for understanding learning from perturbations using a one-hidden-layer network. Our results highlight that various adversarial perturbations, even perturbations of a few pixels, contain sufficient class features for generalization.
  arXiv  Detail & Related papers  (2024-02-16T06:22:44Z)
• Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery [56.172872410834664]
  Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning. We propose a Memory Consistency guided Divide-and-conquer Learning framework (MCDL) Our method outperforms state-of-the-art models by a large margin on both seen and unseen classes of the generic image recognition.
  arXiv  Detail & Related papers  (2024-01-24T09:39:45Z)
• lpNTK: Better Generalisation with Less Data via Sample Interaction During Learning [22.59771349030541]
  We propose a pseudo Neural Tangent Kernel (lpNTK) to take label information into consideration when measuring the interactions between samples. lpNTK helps to understand learning phenomena identified in previous work, specifically the learning difficulty of samples and forgetting events during learning. We show that using lpNTK to identify and remove poisoning training samples does not hurt the generalisation performance of ANNs.
  arXiv  Detail & Related papers  (2024-01-16T20:20:10Z)
• Intra-class Adaptive Augmentation with Neighbor Correction for Deep Metric Learning [99.14132861655223]
  We propose a novel intra-class adaptive augmentation (IAA) framework for deep metric learning. We reasonably estimate intra-class variations for every class and generate adaptive synthetic samples to support hard samples mining. Our method significantly improves and outperforms the state-of-the-art methods on retrieval performances by 3%-6%.
  arXiv  Detail & Related papers  (2022-11-29T14:52:38Z)
• Jo-SRC: A Contrastive Approach for Combating Noisy Labels [58.867237220886885]
  We propose a noise-robust approach named Jo-SRC (Joint Sample Selection and Model Regularization based on Consistency) Specifically, we train the network in a contrastive learning manner. Predictions from two different views of each sample are used to estimate its "likelihood" of being clean or out-of-distribution.
  arXiv  Detail & Related papers  (2021-03-24T07:26:07Z)
• Estimating informativeness of samples with Smooth Unique Information [108.25192785062367]
  We measure how much a sample informs the final weights and how much it informs the function computed by the weights. We give efficient approximations of these quantities using a linearized network. We apply these measures to several problems, such as dataset summarization.
  arXiv  Detail & Related papers  (2021-01-17T10:29:29Z)
• MOCCA: Multi-Layer One-Class ClassificAtion for Anomaly Detection [16.914663209964697]
  We propose our deep learning approach to the anomaly detection problem named Multi-LayerOne-Class Classification (MOCCA) We explicitly leverage the piece-wise nature of deep neural networks by exploiting information extracted at different depths to detect abnormal data instances. We show that our method reaches superior performances compared to the state-of-the-art approaches available in the literature.
  arXiv  Detail & Related papers  (2020-12-09T08:32:56Z)
• ReMarNet: Conjoint Relation and Margin Learning for Small-Sample Image Classification [49.87503122462432]
  We introduce a novel neural network termed Relation-and-Margin learning Network (ReMarNet) Our method assembles two networks of different backbones so as to learn the features that can perform excellently in both of the aforementioned two classification mechanisms. Experiments on four image datasets demonstrate that our approach is effective in learning discriminative features from a small set of labeled samples.
  arXiv  Detail & Related papers  (2020-06-27T13:50:20Z)
• Robust and On-the-fly Dataset Denoising for Image Classification [72.10311040730815]
  On-the-fly Data Denoising (ODD) is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
  arXiv  Detail & Related papers  (2020-03-24T03:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.