Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Dataset

• URL: http://arxiv.org/abs/2506.12698v1
• Date: Sun, 15 Jun 2025 03:12:38 GMT
• Title: Unsupervised Contrastive Learning Using Out-Of-Distribution Data for Long-Tailed Dataset
• Authors: Cuong Manh Hoang, Yeejin Lee, Byeongkeun Kang,
• Abstract summary: This work addresses the task of self-supervised learning on a long-tailed dataset.<n>It aims to learn balanced and well-separated representations for downstream tasks such as image classification.
• Score: 7.400926717561454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work addresses the task of self-supervised learning (SSL) on a long-tailed dataset that aims to learn balanced and well-separated representations for downstream tasks such as image classification. This task is crucial because the real world contains numerous object categories, and their distributions are inherently imbalanced. Towards robust SSL on a class-imbalanced dataset, we investigate leveraging a network trained using unlabeled out-of-distribution (OOD) data that are prevalently available online. We first train a network using both in-domain (ID) and sampled OOD data by back-propagating the proposed pseudo semantic discrimination loss alongside a domain discrimination loss. The OOD data sampling and loss functions are designed to learn a balanced and well-separated embedding space. Subsequently, we further optimize the network on ID data by unsupervised contrastive learning while using the previously trained network as a guiding network. The guiding network is utilized to select positive/negative samples and to control the strengths of attractive/repulsive forces in contrastive learning. We also distil and transfer its embedding space to the training network to maintain balancedness and separability. Through experiments on four publicly available long-tailed datasets, we demonstrate that the proposed method outperforms previous state-of-the-art methods.

Related papers

• Gradient-Regularized Out-of-Distribution Detection [28.542499196417214]
  One of the challenges for neural networks in real-life applications is the overconfident errors these models make when the data is not from the original training distribution. We propose the idea of leveraging the information embedded in the gradient of the loss function during training to enable the network to learn a desired OOD score for each sample. We also develop a novel energy-based sampling method to allow the network to be exposed to more informative OOD samples during the training phase.
  arXiv  Detail & Related papers  (2024-04-18T17:50:23Z)
• Diffusion-Based Neural Network Weights Generation [80.89706112736353]
  D2NWG is a diffusion-based neural network weights generation technique that efficiently produces high-performing weights for transfer learning. Our method extends generative hyper-representation learning to recast the latent diffusion paradigm for neural network weights generation. Our approach is scalable to large architectures such as large language models (LLMs), overcoming the limitations of current parameter generation techniques.
  arXiv  Detail & Related papers  (2024-02-28T08:34:23Z)
• Preventing Arbitrarily High Confidence on Far-Away Data in Point-Estimated Discriminative Neural Networks [28.97655735976179]
  ReLU networks have been shown to almost always yield high confidence predictions when the test data are far away from the training set. We overcome this problem by adding a term to the output of the neural network that corresponds to the logit of an extra class. This technique provably prevents arbitrarily high confidence on far-away test data while maintaining a simple discriminative point-estimate training.
  arXiv  Detail & Related papers  (2023-11-07T03:19:16Z)
• Distribution Shift Matters for Knowledge Distillation with Webly Collected Images [91.66661969598755]
  We propose a novel method dubbed Knowledge Distillation between Different Distributions" (KD$3$) We first dynamically select useful training instances from the webly collected data according to the combined predictions of teacher network and student network. We also build a new contrastive learning block called MixDistribution to generate perturbed data with a new distribution for instance alignment.
  arXiv  Detail & Related papers  (2023-07-21T10:08:58Z)
• On the Effectiveness of Out-of-Distribution Data in Self-Supervised Long-Tail Learning [15.276356824489431]
  We propose Contrastive with Out-of-distribution (OOD) data for Long-Tail learning (COLT) We empirically identify the counter-intuitive usefulness of OOD samples in SSL long-tailed learning. Our method significantly improves the performance of SSL on long-tailed datasets by a large margin.
  arXiv  Detail & Related papers  (2023-06-08T04:32:10Z)
• Does Decentralized Learning with Non-IID Unlabeled Data Benefit from Self Supervision? [51.00034621304361]
  We study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL) We study the effectiveness of contrastive learning algorithms under decentralized learning settings.
  arXiv  Detail & Related papers  (2022-10-20T01:32:41Z)
• Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning [101.28281124670647]
  Open-set semi-supervised learning (open-set SSL) investigates a challenging but practical scenario where out-of-distribution (OOD) samples are contained in the unlabeled data. We propose a novel training mechanism that could effectively exploit the presence of OOD data for enhanced feature learning. Our approach substantially lifts the performance on open-set SSL and outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-08-12T09:14:44Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)
• Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
  Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions. We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
  arXiv  Detail & Related papers  (2020-06-13T08:24:33Z)
• The Utility of Feature Reuse: Transfer Learning in Data-Starved Regimes [6.419457653976053]
  We describe a transfer learning use case for a domain with a data-starved regime. We evaluate the effectiveness of convolutional feature extraction and fine-tuning. We conclude that transfer learning enhances the performance of CNN architectures in data-starved regimes.
  arXiv  Detail & Related papers  (2020-02-29T18:48:58Z)
• DeGAN : Data-Enriching GAN for Retrieving Representative Samples from a Trained Classifier [58.979104709647295]
  We bridge the gap between the abundance of available data and lack of relevant data, for the future learning tasks of a trained network. We use the available data, that may be an imbalanced subset of the original training dataset, or a related domain dataset, to retrieve representative samples. We demonstrate that data from a related domain can be leveraged to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2019-12-27T02:05:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.