RankFeat&RankWeight: Rank-1 Feature/Weight Removal for Out-of-distribution Detection

• URL: http://arxiv.org/abs/2311.13959v2
• Date: Mon, 27 Nov 2023 09:47:10 GMT
• Title: RankFeat&RankWeight: Rank-1 Feature/Weight Removal for Out-of-distribution Detection
• Authors: Yue Song, Nicu Sebe, Wei Wang
• Abstract summary: textttRankFeat achieves emphstate-of-the-art performance and reduces the average false positive rate (FPR95) by 17.90%. We propose textttRankWeight which removes the rank-1 weight from the parameter matrices of a single deep layer.
• Score: 74.48870221803242
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The task of out-of-distribution (OOD) detection is crucial for deploying machine learning models in real-world settings. In this paper, we observe that the singular value distributions of the in-distribution (ID) and OOD features are quite different: the OOD feature matrix tends to have a larger dominant singular value than the ID feature, and the class predictions of OOD samples are largely determined by it. This observation motivates us to propose \texttt{RankFeat}, a simple yet effective \emph{post hoc} approach for OOD detection by removing the rank-1 matrix composed of the largest singular value and the associated singular vectors from the high-level feature. \texttt{RankFeat} achieves \emph{state-of-the-art} performance and reduces the average false positive rate (FPR95) by 17.90\% compared with the previous best method. The success of \texttt{RankFeat} motivates us to investigate whether a similar phenomenon would exist in the parameter matrices of neural networks. We thus propose \texttt{RankWeight} which removes the rank-1 weight from the parameter matrices of a single deep layer. Our \texttt{RankWeight}is also \emph{post hoc} and only requires computing the rank-1 matrix once. As a standalone approach, \texttt{RankWeight} has very competitive performance against other methods across various backbones. Moreover, \texttt{RankWeight} enjoys flexible compatibility with a wide range of OOD detection methods. The combination of \texttt{RankWeight} and \texttt{RankFeat} refreshes the new \emph{state-of-the-art} performance, achieving the FPR95 as low as 16.13\% on the ImageNet-1k benchmark. Extensive ablation studies and comprehensive theoretical analyses are presented to support the empirical results.

Related papers

• SeTAR: Out-of-Distribution Detection with Selective Low-Rank Approximation [5.590633742488972]
  Out-of-distribution (OOD) detection is crucial for the safe deployment of neural networks. We propose SeTAR, a training-free OOD detection method. SeTAR enhances OOD detection via post-hoc modification of the model's weight matrices using a simple greedy search algorithm. Our work offers a scalable, efficient solution for OOD detection, setting a new state-of-the-art in this area.
  arXiv  Detail & Related papers  (2024-06-18T13:55:13Z)
• MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning [45.93128932828256]
  Masked LoRA Experts (MLAE) is an innovative approach that applies the concept of masking to visual PEFT. Our method incorporates a cellular decomposition strategy that transforms a low-rank matrix into independent rank-1 submatrices. We show that MLAE achieves new state-of-the-art (SOTA) performance with an average accuracy score of 78.8% on the VTAB-1k benchmark and 90.9% on the FGVC benchmark.
  arXiv  Detail & Related papers  (2024-05-29T08:57:23Z)
• Model-free Test Time Adaptation for Out-Of-Distribution Detection [62.49795078366206]
  We propose a Non-Parametric Test Time textbfAdaptation framework for textbfDistribution textbfDetection (abbr) abbr utilizes online test samples for model adaptation during testing, enhancing adaptability to changing data distributions. We demonstrate the effectiveness of abbr through comprehensive experiments on multiple OOD detection benchmarks.
  arXiv  Detail & Related papers  (2023-11-28T02:00:47Z)
• Spectral Entry-wise Matrix Estimation for Low-Rank Reinforcement Learning [53.445068584013896]
  We study matrix estimation problems arising in reinforcement learning (RL) with low-rank structure. In low-rank bandits, the matrix to be recovered specifies the expected arm rewards, and for low-rank Markov Decision Processes (MDPs), it may for example characterize the transition kernel of the MDP. We show that simple spectral-based matrix estimation approaches efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error.
  arXiv  Detail & Related papers  (2023-10-10T17:06:41Z)
• Precise Benchmarking of Explainable AI Attribution Methods [0.0]
  We propose a novel evaluation approach for benchmarking state-of-the-art XAI attribution methods. Our proposal consists of a synthetic classification model accompanied by its derived ground truth explanations. Our experimental results provide novel insights into the performance of Guided-Backprop and Smoothgrad XAI methods.
  arXiv  Detail & Related papers  (2023-08-06T17:03:32Z)
• RankFeat: Rank-1 Feature Removal for Out-of-distribution Detection [65.67315418971688]
  textttRankFeat is a simple yet effective textttpost hoc approach for OOD detection. textttRankFeat achieves the emphstate-of-the-art performance and reduces the average false positive rate (FPR95) by 17.90% compared with the previous best method.
  arXiv  Detail & Related papers  (2022-09-18T16:01:31Z)
• Partial and Asymmetric Contrastive Learning for Out-of-Distribution Detection in Long-Tailed Recognition [80.07843757970923]
  We show that existing OOD detection methods suffer from significant performance degradation when the training set is long-tail distributed. We propose Partial and Asymmetric Supervised Contrastive Learning (PASCL), which explicitly encourages the model to distinguish between tail-class in-distribution samples and OOD samples. Our method outperforms previous state-of-the-art method by $1.29%$, $1.45%$, $0.69%$ anomaly detection false positive rate (FPR) and $3.24%$, $4.06%$, $7.89%$ in-distribution
  arXiv  Detail & Related papers  (2022-07-04T01:53:07Z)
• Monarch: Expressive Structured Matrices for Efficient and Accurate Training [64.6871423399431]
  Large neural networks excel in many domains, but they are expensive to train and fine-tune. A popular approach to reduce their compute or memory requirements is to replace dense weight matrices with structured ones. We propose a class of matrices (Monarch) that is hardware-efficient.
  arXiv  Detail & Related papers  (2022-04-01T17:37:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.