Circumventing Outliers of AutoAugment with Knowledge Distillation

• URL: http://arxiv.org/abs/2003.11342v1
• Date: Wed, 25 Mar 2020 11:51:41 GMT
• Title: Circumventing Outliers of AutoAugment with Knowledge Distillation
• Authors: Longhui Wei, An Xiao, Lingxi Xie, Xin Chen, Xiaopeng Zhang, Qi Tian
• Abstract summary: AutoAugment has been a powerful algorithm that improves the accuracy of many vision tasks. This paper delves deep into the working mechanism, and reveals that AutoAugment may remove part of discriminative information from the training image. To relieve the inaccuracy of supervision, we make use of knowledge distillation that refers to the output of a teacher model to guide network training.
• Score: 102.25991455094832
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AutoAugment has been a powerful algorithm that improves the accuracy of many vision tasks, yet it is sensitive to the operator space as well as hyper-parameters, and an improper setting may degenerate network optimization. This paper delves deep into the working mechanism, and reveals that AutoAugment may remove part of discriminative information from the training image and so insisting on the ground-truth label is no longer the best option. To relieve the inaccuracy of supervision, we make use of knowledge distillation that refers to the output of a teacher model to guide network training. Experiments are performed in standard image classification benchmarks, and demonstrate the effectiveness of our approach in suppressing noise of data augmentation and stabilizing training. Upon the cooperation of knowledge distillation and AutoAugment, we claim the new state-of-the-art on ImageNet classification with a top-1 accuracy of 85.8%.

Related papers

• Explainable Light-Weight Deep Learning Pipeline for Improved Drought Stress Identification [0.0]
  Early identification of drought stress in crops is vital for implementing effective mitigation measures and reducing yield loss. Our work proposes a novel deep learning framework for classifying drought stress in potato crops captured by UAVs in natural settings. A key innovation of our work involves the integration of Gradient-Class Activation Mapping (Grad-CAM), an explainability technique.
  arXiv  Detail & Related papers  (2024-04-15T18:26:03Z)
• TSCM: A Teacher-Student Model for Vision Place Recognition Using Cross-Metric Knowledge Distillation [6.856317526681759]
  Visual place recognition plays a pivotal role in autonomous exploration and navigation of mobile robots. Existing methods overcome this by exploiting powerful yet large networks. We propose a high-performance teacher and lightweight student distillation framework called TSCM.
  arXiv  Detail & Related papers  (2024-04-02T02:29:41Z)
• Efficient Visual Fault Detection for Freight Train Braking System via Heterogeneous Self Distillation in the Wild [8.062167870951706]
  This paper proposes a heterogeneous self-distillation framework to ensure detection accuracy and speed. We employ a novel loss function that makes the network easily concentrate on values near the label to improve learning efficiency. Our framework can achieve over 37 frames per second and maintain the highest accuracy in comparison with traditional distillation approaches.
  arXiv  Detail & Related papers  (2023-07-03T01:27:39Z)
• Maximum Likelihood Distillation for Robust Modulation Classification [50.51144496609274]
  We build on knowledge distillation ideas and adversarial training to build more robust AMC systems. We propose to use the Maximum Likelihood function, which could solve the AMC problem in offline settings, to generate better training labels.
  arXiv  Detail & Related papers  (2022-11-01T21:06:11Z)
• Boosting Facial Expression Recognition by A Semi-Supervised Progressive Teacher [54.50747989860957]
  We propose a semi-supervised learning algorithm named Progressive Teacher (PT) to utilize reliable FER datasets as well as large-scale unlabeled expression images for effective training. Experiments on widely-used databases RAF-DB and FERPlus validate the effectiveness of our method, which achieves state-of-the-art performance with accuracy of 89.57% on RAF-DB.
  arXiv  Detail & Related papers  (2022-05-28T07:47:53Z)
• Spirit Distillation: Precise Real-time Prediction with Insufficient Data [4.6247655021017655]
  We propose a new training framework named Spirit Distillation(SD) It extends the ideas of fine-tuning-based transfer learning(FTT) and feature-based knowledge distillation. Results demonstrate the boosting performance in segmentation(mIOU) and high-precision accuracy boost by 1.4% and 8.2% respectively.
  arXiv  Detail & Related papers  (2021-03-25T10:23:30Z)
• Computation-Efficient Knowledge Distillation via Uncertainty-Aware Mixup [91.1317510066954]
  We study a little-explored but important question, i.e., knowledge distillation efficiency. Our goal is to achieve a performance comparable to conventional knowledge distillation with a lower computation cost during training. We show that the UNcertainty-aware mIXup (UNIX) can serve as a clean yet effective solution.
  arXiv  Detail & Related papers  (2020-12-17T06:52:16Z)
• Adversarial Training Reduces Information and Improves Transferability [81.59364510580738]
  Recent results show that features of adversarially trained networks for classification, in addition to being robust, enable desirable properties such as invertibility. We show that the Adversarial Training can improve linear transferability to new tasks, from which arises a new trade-off between transferability of representations and accuracy on the source task.
  arXiv  Detail & Related papers  (2020-07-22T08:30:16Z)
• Knowledge Distillation Meets Self-Supervision [109.6400639148393]
  Knowledge distillation involves extracting "dark knowledge" from a teacher network to guide the learning of a student network. We show that the seemingly different self-supervision task can serve as a simple yet powerful solution. By exploiting the similarity between those self-supervision signals as an auxiliary task, one can effectively transfer the hidden information from the teacher to the student.
  arXiv  Detail & Related papers  (2020-06-12T12:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.