Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning

• URL: http://arxiv.org/abs/2410.11206v1
• Date: Tue, 15 Oct 2024 02:47:57 GMT
• Title: Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning
• Authors: Jingyang Li, Jiachun Pan, Vincent Y. F. Tan, Kim-Chuan Toh, Pan Zhou,
• Abstract summary: Semi-supervised learning (SSL) has shown significant generalization advantages over supervised learning (SL) We present the first theoretical justification for the enhanced test accuracy observed in FixMatch-like SSL applied to deep neural networks (DNNs) We show that our analysis framework can be applied to other FixMatch-like SSL methods, e.g., FlexMatch, FreeMatch, Dash, and SoftMatch.
• Score: 97.1805039692731
• License:
• Abstract: Semi-supervised learning (SSL), exemplified by FixMatch (Sohn et al., 2020), has shown significant generalization advantages over supervised learning (SL), particularly in the context of deep neural networks (DNNs). However, it is still unclear, from a theoretical standpoint, why FixMatch-like SSL algorithms generalize better than SL on DNNs. In this work, we present the first theoretical justification for the enhanced test accuracy observed in FixMatch-like SSL applied to DNNs by taking convolutional neural networks (CNNs) on classification tasks as an example. Our theoretical analysis reveals that the semantic feature learning processes in FixMatch and SL are rather different. In particular, FixMatch learns all the discriminative features of each semantic class, while SL only randomly captures a subset of features due to the well-known lottery ticket hypothesis. Furthermore, we show that our analysis framework can be applied to other FixMatch-like SSL methods, e.g., FlexMatch, FreeMatch, Dash, and SoftMatch. Inspired by our theoretical analysis, we develop an improved variant of FixMatch, termed Semantic-Aware FixMatch (SA-FixMatch). Experimental results corroborate our theoretical findings and the enhanced generalization capability of SA-FixMatch.

Related papers

• AstMatch: Adversarial Self-training Consistency Framework for Semi-Supervised Medical Image Segmentation [19.80612796391153]
  Semi-supervised learning (SSL) has shown considerable potential in medical image segmentation. In this work, we propose an adversarial self-training consistency framework (AstMatch) The proposed AstMatch has been extensively evaluated with cutting-edge SSL methods on three public-available datasets.
  arXiv  Detail & Related papers  (2024-06-28T04:38:12Z)
• Benchmark for Uncertainty & Robustness in Self-Supervised Learning [0.0]
  Self-Supervised Learning is crucial for real-world applications, especially in data-hungry domains such as healthcare and self-driving cars. In this paper, we explore variants of SSL methods, including Jigsaw Puzzles, Context, Rotation, Geometric Transformations Prediction for vision, as well as BERT and GPT for language tasks. Our goal is to create a benchmark with outputs from experiments, providing a starting point for new SSL methods in Reliable Machine Learning.
  arXiv  Detail & Related papers  (2022-12-23T15:46:23Z)
• Flaky Performances when Pretraining on Relational Databases [20.840010074776544]
  We explore the downstream task performances for graph neural network (GNN) self-supervised learning (SSL) methods trained on subgraphs extracted from relational databases (RDBs) We found that naively porting contrastive SSL techniques can cause negative transfer'' Based on the conjecture that contrastive SSL conflicts with the message passing layers of the GNN, we propose InfoNode: a contrastive loss aiming to maximize the mutual information between a node's initial- and final-layer representation.
  arXiv  Detail & Related papers  (2022-11-09T21:50:41Z)
• Improving Self-Supervised Learning by Characterizing Idealized Representations [155.1457170539049]
  We prove necessary and sufficient conditions for any task invariant to given data augmentations. For contrastive learning, our framework prescribes simple but significant improvements to previous methods. For non-contrastive learning, we use our framework to derive a simple and novel objective.
  arXiv  Detail & Related papers  (2022-09-13T18:01:03Z)
• Self-Ensembling GAN for Cross-Domain Semantic Segmentation [107.27377745720243]
  This paper proposes a self-ensembling generative adversarial network (SE-GAN) exploiting cross-domain data for semantic segmentation. In SE-GAN, a teacher network and a student network constitute a self-ensembling model for generating semantic segmentation maps, which together with a discriminator, forms a GAN. Despite its simplicity, we find SE-GAN can significantly boost the performance of adversarial training and enhance the stability of the model.
  arXiv  Detail & Related papers  (2021-12-15T09:50:25Z)
• Understanding self-supervised Learning Dynamics without Contrastive Pairs [72.1743263777693]
  Contrastive approaches to self-supervised learning (SSL) learn representations by minimizing the distance between two augmented views of the same data point. BYOL and SimSiam, show remarkable performance it without negative pairs. We study the nonlinear learning dynamics of non-contrastive SSL in simple linear networks.
  arXiv  Detail & Related papers  (2021-02-12T22:57:28Z)
• Interventional Few-Shot Learning [88.31112565383457]
  We propose a novel Few-Shot Learning paradigm: Interventional Few-Shot Learning. Code is released at https://github.com/yue-zhongqi/ifsl.
  arXiv  Detail & Related papers  (2020-09-28T01:16:54Z)
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence [93.91751021370638]
  Semi-supervised learning (SSL) provides an effective means of leveraging unlabeled data to improve a model's performance. In this paper, we demonstrate the power of a simple combination of two common SSL methods: consistency regularization and pseudo-labeling. Our algorithm, FixMatch, first generates pseudo-labels using the model's predictions on weakly-augmented unlabeled images.
  arXiv  Detail & Related papers  (2020-01-21T18:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.