Out-of-Distributed Semantic Pruning for Robust Semi-Supervised Learning
- URL: http://arxiv.org/abs/2305.18158v2
- Date: Tue, 30 May 2023 03:33:31 GMT
- Title: Out-of-Distributed Semantic Pruning for Robust Semi-Supervised Learning
- Authors: Yu Wang, Pengchong Qiao, Chang Liu, Guoli Song, Xiawu Zheng, Jie Chen
- Abstract summary: We propose a unified framework termed OOD Semantic Pruning (OSP), which aims at pruning OOD semantics out from in-distribution (ID) features.
OSP surpasses the previous state-of-the-art by 13.7% on accuracy for ID classification and 5.9% on AUROC for OOD detection on TinyImageNet dataset.
- Score: 17.409939628100517
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent advances in robust semi-supervised learning (SSL) typically filter
out-of-distribution (OOD) information at the sample level. We argue that an
overlooked problem of robust SSL is its corrupted information on semantic
level, practically limiting the development of the field. In this paper, we
take an initial step to explore and propose a unified framework termed OOD
Semantic Pruning (OSP), which aims at pruning OOD semantics out from
in-distribution (ID) features. Specifically, (i) we propose an aliasing OOD
matching module to pair each ID sample with an OOD sample with semantic
overlap. (ii) We design a soft orthogonality regularization, which first
transforms each ID feature by suppressing its semantic component that is
collinear with paired OOD sample. It then forces the predictions before and
after soft orthogonality decomposition to be consistent. Being practically
simple, our method shows a strong performance in OOD detection and ID
classification on challenging benchmarks. In particular, OSP surpasses the
previous state-of-the-art by 13.7% on accuracy for ID classification and 5.9%
on AUROC for OOD detection on TinyImageNet dataset. The source codes are
publicly available at https://github.com/rain305f/OSP.
Related papers
- Rethinking the Evaluation of Out-of-Distribution Detection: A Sorites Paradox [70.57120710151105]
Most existing out-of-distribution (OOD) detection benchmarks classify samples with novel labels as the OOD data.
Some marginal OOD samples actually have close semantic contents to the in-distribution (ID) sample, which makes determining the OOD sample a Sorites Paradox.
We construct a benchmark named Incremental Shift OOD (IS-OOD) to address the issue.
arXiv Detail & Related papers (2024-06-14T09:27:56Z) - EAT: Towards Long-Tailed Out-of-Distribution Detection [55.380390767978554]
This paper addresses the challenging task of long-tailed OOD detection.
The main difficulty lies in distinguishing OOD data from samples belonging to the tail classes.
We propose two simple ideas: (1) Expanding the in-distribution class space by introducing multiple abstention classes, and (2) Augmenting the context-limited tail classes by overlaying images onto the context-rich OOD data.
arXiv Detail & Related papers (2023-12-14T13:47:13Z) - Classifier-head Informed Feature Masking and Prototype-based Logit
Smoothing for Out-of-Distribution Detection [27.062465089674763]
Out-of-distribution (OOD) detection is essential when deploying neural networks in the real world.
One main challenge is that neural networks often make overconfident predictions on OOD data.
We propose an effective post-hoc OOD detection method based on a new feature masking strategy and a novel logit smoothing strategy.
arXiv Detail & Related papers (2023-10-27T12:42:17Z) - From Global to Local: Multi-scale Out-of-distribution Detection [129.37607313927458]
Out-of-distribution (OOD) detection aims to detect "unknown" data whose labels have not been seen during the in-distribution (ID) training process.
Recent progress in representation learning gives rise to distance-based OOD detection.
We propose Multi-scale OOD DEtection (MODE), a first framework leveraging both global visual information and local region details.
arXiv Detail & Related papers (2023-08-20T11:56:25Z) - Generalized Open-World Semi-Supervised Object Detection [22.058195650206944]
We introduce an ensemble-based OOD Explorer for detection and classification, and an adaptable semi-supervised object detection framework.
We demonstrate that our method performs competitively against state-of-the-art OOD detection algorithms and also significantly boosts the semi-supervised learning performance for both ID and OOD classes.
arXiv Detail & Related papers (2023-07-28T17:59:03Z) - Holistic Sentence Embeddings for Better Out-of-Distribution Detection [12.640837452980332]
We propose a simple embedding approach named Avg-Avg, which averages all token representations from each intermediate layer as the sentence embedding.
Our analysis demonstrates that it indeed helps preserve general linguistic knowledge in fine-tuned PLMs and substantially benefits detecting background shifts.
arXiv Detail & Related papers (2022-10-14T03:22:58Z) - Full-Spectrum Out-of-Distribution Detection [42.98617540431124]
We take into account both shift types and introduce full-spectrum OOD (FS-OOD) detection.
We propose SEM, a simple feature-based semantics score function.
SEM significantly outperforms current state-of-the-art methods.
arXiv Detail & Related papers (2022-04-11T17:59:14Z) - Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for
Open-Set Semi-Supervised Learning [101.28281124670647]
Open-set semi-supervised learning (open-set SSL) investigates a challenging but practical scenario where out-of-distribution (OOD) samples are contained in the unlabeled data.
We propose a novel training mechanism that could effectively exploit the presence of OOD data for enhanced feature learning.
Our approach substantially lifts the performance on open-set SSL and outperforms the state-of-the-art by a large margin.
arXiv Detail & Related papers (2021-08-12T09:14:44Z) - Triggering Failures: Out-Of-Distribution detection by learning from
local adversarial attacks in Semantic Segmentation [76.2621758731288]
We tackle the detection of out-of-distribution (OOD) objects in semantic segmentation.
Our main contribution is a new OOD detection architecture called ObsNet associated with a dedicated training scheme based on Local Adversarial Attacks (LAA)
We show it obtains top performances both in speed and accuracy when compared to ten recent methods of the literature on three different datasets.
arXiv Detail & Related papers (2021-08-03T17:09:56Z) - Multi-Task Curriculum Framework for Open-Set Semi-Supervised Learning [54.85397562961903]
Semi-supervised learning (SSL) has been proposed to leverage unlabeled data for training powerful models when only limited labeled data is available.
We address a more complex novel scenario named open-set SSL, where out-of-distribution (OOD) samples are contained in unlabeled data.
Our method achieves state-of-the-art results by successfully eliminating the effect of OOD samples.
arXiv Detail & Related papers (2020-07-22T10:33:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.