Out-of-Distributed Semantic Pruning for Robust Semi-Supervised Learning

• URL: http://arxiv.org/abs/2305.18158v2
• Date: Tue, 30 May 2023 03:33:31 GMT
• Title: Out-of-Distributed Semantic Pruning for Robust Semi-Supervised Learning
• Authors: Yu Wang, Pengchong Qiao, Chang Liu, Guoli Song, Xiawu Zheng, Jie Chen
• Abstract summary: We propose a unified framework termed OOD Semantic Pruning (OSP), which aims at pruning OOD semantics out from in-distribution (ID) features. OSP surpasses the previous state-of-the-art by 13.7% on accuracy for ID classification and 5.9% on AUROC for OOD detection on TinyImageNet dataset.
• Score: 17.409939628100517
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in robust semi-supervised learning (SSL) typically filter out-of-distribution (OOD) information at the sample level. We argue that an overlooked problem of robust SSL is its corrupted information on semantic level, practically limiting the development of the field. In this paper, we take an initial step to explore and propose a unified framework termed OOD Semantic Pruning (OSP), which aims at pruning OOD semantics out from in-distribution (ID) features. Specifically, (i) we propose an aliasing OOD matching module to pair each ID sample with an OOD sample with semantic overlap. (ii) We design a soft orthogonality regularization, which first transforms each ID feature by suppressing its semantic component that is collinear with paired OOD sample. It then forces the predictions before and after soft orthogonality decomposition to be consistent. Being practically simple, our method shows a strong performance in OOD detection and ID classification on challenging benchmarks. In particular, OSP surpasses the previous state-of-the-art by 13.7% on accuracy for ID classification and 5.9% on AUROC for OOD detection on TinyImageNet dataset. The source codes are publicly available at https://github.com/rain305f/OSP.

Related papers

• Semantic or Covariate? A Study on the Intractable Case of Out-of-Distribution Detection [70.57120710151105]
  We provide a more precise definition of the Semantic Space for the ID distribution. We also define the "Tractable OOD" setting which ensures the distinguishability of OOD and ID distributions.
  arXiv  Detail & Related papers  (2024-11-18T03:09:39Z)
• Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox [70.57120710151105]
  Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data. Some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox. We construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue.
  arXiv  Detail & Related papers  (2024-06-14T09:27:56Z)
• Classifier-head Informed Feature Masking and Prototype-based Logit Smoothing for Out-of-Distribution Detection [27.062465089674763]
  Out-of-distribution (OOD) detection is essential when deploying neural networks in the real world. One main challenge is that neural networks often make overconfident predictions on OOD data. We propose an effective post-hoc OOD detection method based on a new feature masking strategy and a novel logit smoothing strategy.
  arXiv  Detail & Related papers  (2023-10-27T12:42:17Z)
• From Global to Local: Multi-scale Out-of-distribution Detection [129.37607313927458]
  Out-of-distribution (OOD) detection aims to detect "unknown" data whose labels have not been seen during the in-distribution (ID) training process. Recent progress in representation learning gives rise to distance-based OOD detection. We propose Multi-scale OOD DEtection (MODE), a first framework leveraging both global visual information and local region details.
  arXiv  Detail & Related papers  (2023-08-20T11:56:25Z)
• Generalized Open-World Semi-Supervised Object Detection [22.058195650206944]
  We introduce an ensemble-based OOD Explorer for detection and classification, and an adaptable semi-supervised object detection framework. We demonstrate that our method performs competitively against state-of-the-art OOD detection algorithms and also significantly boosts the semi-supervised learning performance for both ID and OOD classes.
  arXiv  Detail & Related papers  (2023-07-28T17:59:03Z)
• Holistic Sentence Embeddings for Better Out-of-Distribution Detection [12.640837452980332]
  We propose a simple embedding approach named Avg-Avg, which averages all token representations from each intermediate layer as the sentence embedding. Our analysis demonstrates that it indeed helps preserve general linguistic knowledge in fine-tuned PLMs and substantially benefits detecting background shifts.
  arXiv  Detail & Related papers  (2022-10-14T03:22:58Z)
• Full-Spectrum Out-of-Distribution Detection [42.98617540431124]
  We take into account both shift types and introduce full-spectrum OOD (FS-OOD) detection. We propose SEM, a simple feature-based semantics score function. SEM significantly outperforms current state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-04-11T17:59:14Z)
• Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning [101.28281124670647]
  Open-set semi-supervised learning (open-set SSL) investigates a challenging but practical scenario where out-of-distribution (OOD) samples are contained in the unlabeled data. We propose a novel training mechanism that could effectively exploit the presence of OOD data for enhanced feature learning. Our approach substantially lifts the performance on open-set SSL and outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-08-12T09:14:44Z)
• Triggering Failures: Out-Of-Distribution detection by learning from local adversarial attacks in Semantic Segmentation [76.2621758731288]
  We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation. Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA) We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
  arXiv  Detail & Related papers  (2021-08-03T17:09:56Z)
• Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning [54.85397562961903]
  Semi-supervised learning (SSL) has been proposed to leverage unlabeled data for training powerful models when only limited labeled data is available. We address a more complex novel scenario named open-set SSL, where out-of-distribution (OOD) samples are contained in unlabeled data. Our method achieves state-of-the-art results by successfully eliminating the effect of OOD samples.
  arXiv  Detail & Related papers  (2020-07-22T10:33:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.