OOD Detection with immature Models

• URL: http://arxiv.org/abs/2502.00820v1
• Date: Sun, 02 Feb 2025 15:14:17 GMT
• Title: OOD Detection with immature Models
• Authors: Behrooz Montazeran, Ullrich Köthe,
• Abstract summary: Likelihood-based deep generative models (DGMs) have gained significant attention for their ability to approximate the distributions of high-dimensional data. These models lack a performance guarantee in assigning higher likelihood values to in-distribution (ID) inputs, data the models are trained on, compared to out-of-distribution (OOD) inputs. In this work, we demonstrate that using immature models,stopped at early stages of training, can mostly achieve equivalent or even superior results on this downstream task.
• Score: 8.477943884416023
• License:
• Abstract: Likelihood-based deep generative models (DGMs) have gained significant attention for their ability to approximate the distributions of high-dimensional data. However, these models lack a performance guarantee in assigning higher likelihood values to in-distribution (ID) inputs, data the models are trained on, compared to out-of-distribution (OOD) inputs. This counter-intuitive behaviour is particularly pronounced when ID inputs are more complex than OOD data points. One potential approach to address this challenge involves leveraging the gradient of a data point with respect to the parameters of the DGMs. A recent OOD detection framework proposed estimating the joint density of layer-wise gradient norms for a given data point as a model-agnostic method, demonstrating superior performance compared to the Typicality Test across likelihood-based DGMs and image dataset pairs. In particular, most existing methods presuppose access to fully converged models, the training of which is both time-intensive and computationally demanding. In this work, we demonstrate that using immature models,stopped at early stages of training, can mostly achieve equivalent or even superior results on this downstream task compared to mature models capable of generating high-quality samples that closely resemble ID data. This novel finding enhances our understanding of how DGMs learn the distribution of ID data and highlights the potential of leveraging partially trained models for downstream tasks. Furthermore, we offer a possible explanation for this unexpected behaviour through the concept of support overlap.

Related papers

• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• Sub-graph Based Diffusion Model for Link Prediction [43.15741675617231]
  Denoising Diffusion Probabilistic Models (DDPMs) represent a contemporary class of generative models with exceptional qualities. We build a novel generative model for link prediction using a dedicated design to decompose the likelihood estimation process via the Bayesian formula. Our proposed method presents numerous advantages: (1) transferability across datasets without retraining, (2) promising generalization on limited training data, and (3) robustness against graph adversarial attacks.
  arXiv  Detail & Related papers  (2024-09-13T02:23:55Z)
• Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification [49.09505771145326]
  We propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels. Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
  arXiv  Detail & Related papers  (2024-04-26T06:00:27Z)
• DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception [78.26734070960886]
  Current perceptive models heavily depend on resource-intensive datasets. We introduce perception-aware loss (P.A. loss) through segmentation, improving both quality and controllability. Our method customizes data augmentation by extracting and utilizing perception-aware attribute (P.A. Attr) during generation.
  arXiv  Detail & Related papers  (2024-03-20T04:58:03Z)
• Approximations to the Fisher Information Metric of Deep Generative Models for Out-Of-Distribution Detection [2.3749120526936465]
  We show that deep generative models consistently infer higher log-likelihoods for OOD data than data they were trained on. We use the gradient of a data point with respect to the parameters of the deep generative model for OOD detection, based on the simple intuition that OOD data should have larger gradient norms than training data. Our empirical results indicate that this method outperforms the Typicality test for most deep generative models and image dataset pairings.
  arXiv  Detail & Related papers  (2024-03-03T11:36:35Z)
• Unleashing Mask: Explore the Intrinsic Out-of-Distribution Detection Capability [70.72426887518517]
  Out-of-distribution (OOD) detection is an indispensable aspect of secure AI when deploying machine learning models in real-world applications. We propose a novel method, Unleashing Mask, which aims to restore the OOD discriminative capabilities of the well-trained model with ID data. Our method utilizes a mask to figure out the memorized atypical samples, and then finetune the model or prune it with the introduced mask to forget them.
  arXiv  Detail & Related papers  (2023-06-06T14:23:34Z)
• Watermarking for Out-of-distribution Detection [76.20630986010114]
  Out-of-distribution (OOD) detection aims to identify OOD data based on representations extracted from well-trained deep models. We propose a general methodology named watermarking in this paper. We learn a unified pattern that is superimposed onto features of original data, and the model's detection capability is largely boosted after watermarking.
  arXiv  Detail & Related papers  (2022-10-27T06:12:32Z)
• Robust Out-of-Distribution Detection on Deep Probabilistic Generative Models [0.06372261626436676]
  Out-of-distribution (OOD) detection is an important task in machine learning systems. Deep probabilistic generative models facilitate OOD detection by estimating the likelihood of a data sample. We propose a new detection metric that operates without outlier exposure.
  arXiv  Detail & Related papers  (2021-06-15T06:36:10Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)
• Hierarchical VAEs Know What They Don't Know [6.649455007186671]
  We develop a fast, scalable and fully unsupervised likelihood-ratio score for OOD detection. We achieve state-of-the-art results on out-of-distribution detection.
  arXiv  Detail & Related papers  (2021-02-16T16:08:04Z)
• Bigeminal Priors Variational auto-encoder [5.430048915427229]
  Variational auto-encoders (VAEs) are an influential and generally-used class of likelihood-based generative models in unsupervised learning. We introduce a new model, namely Bigeminal Priors Variational auto-encoder (BPVAE), to address this phenomenon. BPVAE learns two datasets' features, assigning a higher likelihood for the training dataset than the simple dataset.
  arXiv  Detail & Related papers  (2020-10-05T07:10:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.