An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training

• URL: http://arxiv.org/abs/2010.13116v1
• Date: Sun, 25 Oct 2020 13:35:23 GMT
• Title: An empirical study of domain-agnostic semi-supervised learning via energy-based models: joint-training and pre-training
• Authors: Yunfu Song, Huahuan Zheng, Zhijian Ou
• Abstract summary: generative SSL methods involve unsupervised learning based on generative models by either joint-training or pre-training. Joint-training estimates the joint distribution of observations and labels, while pre-training is taken over observations only. It is found that joint-training EBMs outperform pre-training EBMs marginally but nearly consistently.
• Score: 16.14838937433809
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A class of recent semi-supervised learning (SSL) methods heavily rely on domain-specific data augmentations. In contrast, generative SSL methods involve unsupervised learning based on generative models by either joint-training or pre-training, and are more appealing from the perspective of being domain-agnostic, since they do not inherently require data augmentations. Joint-training estimates the joint distribution of observations and labels, while pre-training is taken over observations only. Recently, energy-based models (EBMs) have achieved promising results for generative modeling. Joint-training via EBMs for SSL has been explored with encouraging results across different data modalities. In this paper, we make two contributions. First, we explore pre-training via EBMs for SSL and compare it to joint-training. Second, a suite of experiments are conducted over domains of image classification and natural language labeling to give a realistic whole picture of the performances of EBM based SSL methods. It is found that joint-training EBMs outperform pre-training EBMs marginally but nearly consistently.

Related papers

• Joint-stochastic-approximation Random Fields with Application to Semi-supervised Learning [10.293017518216908]
  We present joint-stochastic-approximation random fields (JRFs) -- a new family of algorithms for building deep undirected generative models.<n>It is found through synthetic experiments that JRFs work well in balancing mode covering and mode missing, and match the empirical data distribution well.
  arXiv  Detail & Related papers  (2025-05-24T07:04:32Z)
• Revisiting semi-supervised learning in the era of foundation models [28.414667991336067]
  Semi-supervised learning (SSL) leverages abundant unlabeled data alongside limited labeled data to enhance learning. We develop new SSL benchmark datasets where frozen vision foundation models (VFMs) underperform and systematically evaluate representative SSL methods. We make a surprising observation: parameter-efficient fine-tuning (PEFT) using only labeled data often matches SSL performance, even without leveraging unlabeled data. To overcome the notorious issue of noisy pseudo-labels, we propose ensembling multiple PEFT approaches and VFM backbones to produce more robust pseudo-labels.
  arXiv  Detail & Related papers  (2025-03-12T18:01:10Z)
• A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
  Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
  arXiv  Detail & Related papers  (2024-07-16T23:17:36Z)
• ItTakesTwo: Leveraging Peer Representations for Semi-supervised LiDAR Semantic Segmentation [24.743048965822297]
  This paper introduces a novel semi-supervised LiDAR semantic segmentation framework called ItTakesTwo (IT2) IT2 is designed to ensure consistent predictions from peer LiDAR representations, thereby improving the perturbation effectiveness in consistency learning. Results on public benchmarks show that our approach achieves remarkable improvements over the previous state-of-the-art (SOTA) methods in the field.
  arXiv  Detail & Related papers  (2024-07-09T18:26:53Z)
• Using Self-supervised Learning Can Improve Model Fairness [10.028637666224093]
  Self-supervised learning (SSL) has become the de facto training paradigm of large models. This study explores the impact of pre-training and fine-tuning strategies on fairness. We introduce a fairness assessment framework for SSL, comprising five stages: defining dataset requirements, pre-training, fine-tuning with gradual unfreezing, assessing representation similarity conditioned on demographics, and establishing domain-specific evaluation processes.
  arXiv  Detail & Related papers  (2024-06-04T14:38:30Z)
• Evaluating Fairness in Self-supervised and Supervised Models for Sequential Data [10.626503137418636]
  Self-supervised learning (SSL) has become the de facto training paradigm of large models. This study explores the impact of pre-training and fine-tuning strategies on fairness.
  arXiv  Detail & Related papers  (2024-01-03T09:31:43Z)
• Self-supervised learning for skin cancer diagnosis with limited training data [0.196629787330046]
  Self-supervised learning (SSL) is an alternative to the standard supervised pre-training on ImageNet for scenarios with limited training data. We consider textitfurther SSL pre-training on task-specific datasets, where our implementation is motivated by supervised transfer learning. We find minimal further SSL pre-training on task-specific data can be as effective as large-scale SSL pre-training on ImageNet for medical image classification tasks with limited labelled data.
  arXiv  Detail & Related papers  (2024-01-01T08:11:38Z)
• Learning Energy-Based Models by Cooperative Diffusion Recovery Likelihood [64.95663299945171]
  Training energy-based models (EBMs) on high-dimensional data can be both challenging and time-consuming. There exists a noticeable gap in sample quality between EBMs and other generative frameworks like GANs and diffusion models. We propose cooperative diffusion recovery likelihood (CDRL), an effective approach to tractably learn and sample from a series of EBMs.
  arXiv  Detail & Related papers  (2023-09-10T22:05:24Z)
• Rethinking Self-Supervised Visual Representation Learning in Pre-training for 3D Human Pose and Shape Estimation [57.206129938611454]
  Self-supervised representation learning (SSL) methods have outperformed the ImageNet classification pre-training for vision tasks such as object detection. We empirically study and analyze the effects of SSL and compare it with other pre-training alternatives for 3DHPSE. Our observations challenge the naive application of the current SSL pre-training to 3DHPSE and relight the value of other data types in the pre-training aspect.
  arXiv  Detail & Related papers  (2023-03-09T16:17:52Z)
• How robust are pre-trained models to distribution shift? [82.08946007821184]
  We show how spurious correlations affect the performance of popular self-supervised learning (SSL) and auto-encoder based models (AE) We develop a novel evaluation scheme with the linear head trained on out-of-distribution (OOD) data, to isolate the performance of the pre-trained models from a potential bias of the linear head used for evaluation.
  arXiv  Detail & Related papers  (2022-06-17T16:18:28Z)
• Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning [101.28281124670647]
  Open-set semi-supervised learning (open-set SSL) investigates a challenging but practical scenario where out-of-distribution (OOD) samples are contained in the unlabeled data. We propose a novel training mechanism that could effectively exploit the presence of OOD data for enhanced feature learning. Our approach substantially lifts the performance on open-set SSL and outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-08-12T09:14:44Z)
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning [77.57285768500225]
  Recently proposed consistency-based Semi-Supervised Learning (SSL) methods have advanced the state of the art in several SSL tasks. Despite these advances, the understanding of these methods is still relatively limited.
  arXiv  Detail & Related papers  (2021-01-18T10:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.