Unleashing the Potential of Unsupervised Deep Outlier Detection through Automated Training Stopping

• URL: http://arxiv.org/abs/2305.16777v1
• Date: Fri, 26 May 2023 09:39:36 GMT
• Title: Unleashing the Potential of Unsupervised Deep Outlier Detection through Automated Training Stopping
• Authors: Yihong Huang, Yuang Zhang, Liping Wang, Xuemin Lin
• Abstract summary: Outlier detection (OD) has received continuous research interests due to its wide applications. We propose a novel metric called loss entropy to internally evaluate the model performance during training. Our approach is the first to enable reliable identification of the optimal training during training without requiring any labels.
• Score: 33.99128209697431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Outlier detection (OD) has received continuous research interests due to its wide applications. With the development of deep learning, increasingly deep OD algorithms are proposed. Despite the availability of numerous deep OD models, existing research has reported that the performance of deep models is extremely sensitive to the configuration of hyperparameters (HPs). However, the selection of HPs for deep OD models remains a notoriously difficult task due to the lack of any labels and long list of HPs. In our study. we shed light on an essential factor, training time, that can introduce significant variation in the performance of deep model. Even the performance is stable across other HPs, training time itself can cause a serious HP sensitivity issue. Motivated by this finding, we are dedicated to formulating a strategy to terminate model training at the optimal iteration. Specifically, we propose a novel metric called loss entropy to internally evaluate the model performance during training while an automated training stopping algorithm is devised. To our knowledge, our approach is the first to enable reliable identification of the optimal training iteration during training without requiring any labels. Our experiments on tabular, image datasets show that our approach can be applied to diverse deep models and datasets. It not only enhances the robustness of deep models to their HPs, but also improves the performance and reduces plenty of training time compared to naive training.

Related papers

• Zero-shot Retrieval: Augmenting Pre-trained Models with Search Engines [83.65380507372483]
  Large pre-trained models can dramatically reduce the amount of task-specific data required to solve a problem, but they often fail to capture domain-specific nuances out of the box. This paper shows how to leverage recent advances in NLP and multi-modal learning to augment a pre-trained model with search engine retrieval.
  arXiv  Detail & Related papers  (2023-11-29T05:33:28Z)
• Continual Learning of Unsupervised Monocular Depth from Videos [19.43053045216986]
  We introduce a framework that captures challenges of continual unsupervised depth estimation (CUDE) We propose a rehearsal-based dual-memory method, MonoDepthCL, which utilizes collected ontemporal consistency for continual learning in depth estimation.
  arXiv  Detail & Related papers  (2023-11-04T12:36:07Z)
• Fast Unsupervised Deep Outlier Model Selection with Hypernetworks [32.15262629879272]
  We introduce HYPER for tuning DOD models, tackling two fundamental challenges: validation without supervision, and efficient search of the HP/model space. A key idea is to design and train a novel hypernetwork (HN) that maps HPs onto optimal weights of the main DOD model. In turn, HYPER capitalizes on a single HN that can dynamically generate weights for many DOD models.
  arXiv  Detail & Related papers  (2023-07-20T02:07:20Z)
• Design Automation for Fast, Lightweight, and Effective Deep Learning Models: A Survey [53.258091735278875]
  This survey covers studies of design automation techniques for deep learning models targeting edge computing. It offers an overview and comparison of key metrics that are used commonly to quantify the proficiency of models in terms of effectiveness, lightness, and computational costs. The survey proceeds to cover three categories of the state-of-the-art of deep model design automation techniques.
  arXiv  Detail & Related papers  (2022-08-22T12:12:43Z)
• Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solution [21.130842136324528]
  We conduct the first large-scale analysis on the HP sensitivity of deep OD methods. We design a HP-robust and scalable deep hyper-ensemble model called ROBOD that assembles models with varying HP configurations.
  arXiv  Detail & Related papers  (2022-06-15T16:46:00Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• Recursive Least-Squares Estimator-Aided Online Learning for Visual Tracking [58.14267480293575]
  We propose a simple yet effective online learning approach for few-shot online adaptation without requiring offline training. It allows an in-built memory retention mechanism for the model to remember the knowledge about the object seen before. We evaluate our approach based on two networks in the online learning families for tracking, i.e., multi-layer perceptrons in RT-MDNet and convolutional neural networks in DiMP.
  arXiv  Detail & Related papers  (2021-12-28T06:51:18Z)
• Consistency Training of Multi-exit Architectures for Sensor Data [0.07614628596146598]
  We present a novel and architecture-agnostic approach for robust training of multi-exit architectures termed consistent exit training. We leverage weak supervision to align model output with consistency training and jointly optimize dual-losses in a multi-task learning fashion over the exits in a network.
  arXiv  Detail & Related papers  (2021-09-27T17:11:25Z)
• Multi-Scale Aligned Distillation for Low-Resolution Detection [68.96325141432078]
  This paper focuses on boosting the performance of low-resolution models by distilling knowledge from a high- or multi-resolution model. On several instance-level detection tasks and datasets, the low-resolution models trained via our approach perform competitively with high-resolution models trained via conventional multi-scale training.
  arXiv  Detail & Related papers  (2021-09-14T12:53:35Z)
• A Simple Fine-tuning Is All You Need: Towards Robust Deep Learning Via Adversarial Fine-tuning [90.44219200633286]
  We propose a simple yet very effective adversarial fine-tuning approach based on a $textitslow start, fast decay$ learning rate scheduling strategy. Experimental results show that the proposed adversarial fine-tuning approach outperforms the state-of-the-art methods on CIFAR-10, CIFAR-100 and ImageNet datasets.
  arXiv  Detail & Related papers  (2020-12-25T20:50:15Z)
• GOAT: GPU Outsourcing of Deep Learning Training With Asynchronous Probabilistic Integrity Verification Inside Trusted Execution Environment [0.0]
  Machine learning models based on Deep Neural Networks (DNNs) are increasingly deployed in a range of applications ranging from self-driving cars to COVID-19 treatment discovery. To support the computational power necessary to learn a DNN, cloud environments with dedicated hardware support have emerged as critical infrastructure. Various approaches have been developed to address these challenges, building on trusted execution environments (TEE)
  arXiv  Detail & Related papers  (2020-10-17T20:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.