Unlocking the Potential of Reverse Distillation for Anomaly Detection

• URL: http://arxiv.org/abs/2412.07579v1
• Date: Tue, 10 Dec 2024 15:14:09 GMT
• Title: Unlocking the Potential of Reverse Distillation for Anomaly Detection
• Authors: Xinyue Liu, Jianyuan Wang, Biao Leng, Shuo Zhang,
• Abstract summary: We propose Expert-Teacher-Student network for simultaneous distillation of both the teacher encoder and student decoder. The added expert network enhances the student's ability to generate normal features and optimize the teacher's differentiation between normal and abnormal features. Our method outperforms existing unsupervised AD methods under Reverse Distillation paradigm.
• Score: 15.89869857998053
• License:
• Abstract: Knowledge Distillation (KD) is a promising approach for unsupervised Anomaly Detection (AD). However, the student network's over-generalization often diminishes the crucial representation differences between teacher and student in anomalous regions, leading to detection failures. To addresses this problem, the widely accepted Reverse Distillation (RD) paradigm designs the asymmetry teacher and student, using an encoder as teacher and a decoder as student. Yet, the design of RD does not ensure that the teacher encoder effectively distinguishes between normal and abnormal features or that the student decoder generates anomaly-free features. Additionally, the absence of skip connections results in a loss of fine details during feature reconstruction. To address these issues, we propose RD with Expert, which introduces a novel Expert-Teacher-Student network for simultaneous distillation of both the teacher encoder and student decoder. The added expert network enhances the student's ability to generate normal features and optimizes the teacher's differentiation between normal and abnormal features, reducing missed detections. Additionally, Guided Information Injection is designed to filter and transfer features from teacher to student, improving detail reconstruction and minimizing false positives. Experiments on several benchmarks prove that our method outperforms existing unsupervised AD methods under RD paradigm, fully unlocking RD's potential.

Related papers

• Dual-Modeling Decouple Distillation for Unsupervised Anomaly Detection [15.89869857998053]
  Over-generalization of the student network to the teacher network may lead to negligible differences in representation capabilities of anomaly. Existing methods address the possible over-generalization by using differentiated students and teachers from the structural perspective. We propose Dual-Modeling Decouple Distillation (DMDD) for the unsupervised anomaly detection.
  arXiv  Detail & Related papers  (2024-08-07T16:39:16Z)
• Multi-Granularity Semantic Revision for Large Language Model Distillation [66.03746866578274]
  We propose a multi-granularity semantic revision method for LLM distillation. At the sequence level, we propose a sequence correction and re-generation strategy. At the token level, we design a distribution adaptive clipping Kullback-Leibler loss as the distillation objective function. At the span level, we leverage the span priors of a sequence to compute the probability correlations within spans, and constrain the teacher and student's probability correlations to be consistent.
  arXiv  Detail & Related papers  (2024-07-14T03:51:49Z)
• Relative Difficulty Distillation for Semantic Segmentation [54.76143187709987]
  We propose a pixel-level KD paradigm for semantic segmentation named Relative Difficulty Distillation (RDD) RDD allows the teacher network to provide effective guidance on learning focus without additional optimization goals. Our research showcases that RDD can integrate with existing KD methods to improve their upper performance bound.
  arXiv  Detail & Related papers  (2024-07-04T08:08:25Z)
• Advancing Pre-trained Teacher: Towards Robust Feature Discrepancy for Anomaly Detection [19.099643719358692]
  We propose a simple yet effective two-stage industrial anomaly detection framework, termed as AAND. In the first anomaly amplification stage, we propose a novel Residual Anomaly Amplification (RAA) module to advance the pre-trained teacher encoder. We further employ a reverse distillation paradigm to train a student decoder, in which a novel Hard Knowledge Distillation (HKD) loss is built to better facilitate the reconstruction of normal patterns.
  arXiv  Detail & Related papers  (2024-05-03T13:00:22Z)
• SD-DiT: Unleashing the Power of Self-supervised Discrimination in Diffusion Transformer [102.39050180060913]
  Diffusion Transformer (DiT) has emerged as the new trend of generative diffusion models on image generation. Recent breakthroughs have been driven by mask strategy that significantly improves the training efficiency of DiT with additional intra-image contextual learning. In this work, we address these limitations by novelly unleashing the self-supervised discrimination knowledge to boost DiT training.
  arXiv  Detail & Related papers  (2024-03-25T17:59:35Z)
• Part Representation Learning with Teacher-Student Decoder for Occluded Person Re-identification [65.63180725319906]
  We propose a Teacher-Student Decoder (TSD) framework for occluded person ReID. Our proposed TSD consists of a Parsing-aware Teacher Decoder (PTD) and a Standard Student Decoder (SSD)
  arXiv  Detail & Related papers  (2023-12-15T13:54:48Z)
• Knowledge Diffusion for Distillation [53.908314960324915]
  The representation gap between teacher and student is an emerging topic in knowledge distillation (KD) We state that the essence of these methods is to discard the noisy information and distill the valuable information in the feature. We propose a novel KD method dubbed DiffKD, to explicitly denoise and match features using diffusion models.
  arXiv  Detail & Related papers  (2023-05-25T04:49:34Z)
• Exploring Inconsistent Knowledge Distillation for Object Detection with Data Augmentation [66.25738680429463]
  Knowledge Distillation (KD) for object detection aims to train a compact detector by transferring knowledge from a teacher model. We propose inconsistent knowledge distillation (IKD) which aims to distill knowledge inherent in the teacher model's counter-intuitive perceptions. Our method outperforms state-of-the-art KD baselines on one-stage, two-stage and anchor-free object detectors.
  arXiv  Detail & Related papers  (2022-09-20T16:36:28Z)
• Anomaly Detection via Reverse Distillation from One-Class Embedding [2.715884199292287]
  We propose a novel T-S model consisting of a teacher encoder and a student decoder. Instead of receiving raw images directly, the student network takes teacher model's one-class embedding as input. In addition, we introduce a trainable one-class bottleneck embedding module in our T-S model.
  arXiv  Detail & Related papers  (2022-01-26T01:48:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.