Improving memory banks for unsupervised learning with large mini-batch, consistency and hard negative mining

• URL: http://arxiv.org/abs/2102.04442v1
• Date: Mon, 8 Feb 2021 18:56:33 GMT
• Title: Improving memory banks for unsupervised learning with large mini-batch, consistency and hard negative mining
• Authors: Adrian Bulat and Enrique S\'anchez-Lozano and Georgios Tzimiropoulos
• Abstract summary: We introduce 3 improvements to the vanilla memory bank-based formulation which brings massive accuracy gains. We enforce the logits obtained by different augmentations of the same sample to be close without trying to enforce discrimination with respect to negative samples. Since instance discrimination is not meaningful for samples that are too visually similar, we devise a novel nearest neighbour approach for improving the memory bank.
• Score: 61.223064077782645
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An important component of unsupervised learning by instance-based discrimination is a memory bank for storing a feature representation for each training sample in the dataset. In this paper, we introduce 3 improvements to the vanilla memory bank-based formulation which brings massive accuracy gains: (a) Large mini-batch: we pull multiple augmentations for each sample within the same batch and show that this leads to better models and enhanced memory bank updates. (b) Consistency: we enforce the logits obtained by different augmentations of the same sample to be close without trying to enforce discrimination with respect to negative samples as proposed by previous approaches. (c) Hard negative mining: since instance discrimination is not meaningful for samples that are too visually similar, we devise a novel nearest neighbour approach for improving the memory bank that gradually merges extremely similar data samples that were previously forced to be apart by the instance level classification loss. Overall, our approach greatly improves the vanilla memory-bank based instance discrimination and outperforms all existing methods for both seen and unseen testing categories with cosine similarity.

Related papers

• Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery [56.172872410834664]
  Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning. We propose a Memory Consistency guided Divide-and-conquer Learning framework (MCDL) Our method outperforms state-of-the-art models by a large margin on both seen and unseen classes of the generic image recognition.
  arXiv  Detail & Related papers  (2024-01-24T09:39:45Z)
• Shrinking Class Space for Enhanced Certainty in Semi-Supervised Learning [59.44422468242455]
  We propose a novel method dubbed ShrinkMatch to learn uncertain samples. For each uncertain sample, it adaptively seeks a shrunk class space, which merely contains the original top-1 class. We then impose a consistency regularization between a pair of strongly and weakly augmented samples in the shrunk space to strive for discriminative representations.
  arXiv  Detail & Related papers  (2023-08-13T14:05:24Z)
• Augmentation Matters: A Simple-yet-Effective Approach to Semi-supervised Semantic Segmentation [46.441263436298996]
  We propose a simple and clean approach that focuses mainly on data perturbations to boost the SSS performance. We adopt a simplified intensity-based augmentation that selects a random number of data transformations. We also randomly inject labelled information to augment the unlabeled samples in an adaptive manner.
  arXiv  Detail & Related papers  (2022-12-09T16:36:52Z)
• Rethinking Clustering-Based Pseudo-Labeling for Unsupervised Meta-Learning [146.11600461034746]
  Method for unsupervised meta-learning, CACTUs, is a clustering-based approach with pseudo-labeling. This approach is model-agnostic and can be combined with supervised algorithms to learn from unlabeled data. We prove that the core reason for this is lack of a clustering-friendly property in the embedding space.
  arXiv  Detail & Related papers  (2022-09-27T19:04:36Z)
• Memory Population in Continual Learning via Outlier Elimination [25.511380924335207]
  Catastrophic forgetting, the phenomenon of forgetting previously learned tasks when learning a new one, is a major hurdle in developing continual learning algorithms. A popular method to alleviate forgetting is to use a memory buffer, which stores a subset of previously learned task examples for use during training on new tasks. This paper introduces Memory Outlier Elimination (MOE), a method for identifying and eliminating outliers in the memory buffer by choosing samples from label-homogeneous subpopulations.
  arXiv  Detail & Related papers  (2022-07-04T00:09:33Z)
• Memory Efficient Class-Incremental Learning for Image Classification [23.666101443616995]
  Class-incremental learning (CIL) usually suffers from the "catastrophic forgetting" problem when updating the joint classification model on the arrival of newly added classes. We propose to keep more auxiliary low-fidelity exemplar samples rather than the original real high-fidelity exemplar samples.
  arXiv  Detail & Related papers  (2020-08-04T08:39:40Z)
• Learning from Aggregate Observations [82.44304647051243]
  We study the problem of learning from aggregate observations where supervision signals are given to sets of instances. We present a general probabilistic framework that accommodates a variety of aggregate observations. Simple maximum likelihood solutions can be applied to various differentiable models.
  arXiv  Detail & Related papers  (2020-04-14T06:18:50Z)
• Robust and On-the-fly Dataset Denoising for Image Classification [72.10311040730815]
  On-the-fly Data Denoising (ODD) is robust to mislabeled examples, while introducing almost zero computational overhead compared to standard training. ODD is able to achieve state-of-the-art results on a wide range of datasets including real-world ones such as WebVision and Clothing1M.
  arXiv  Detail & Related papers  (2020-03-24T03:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.