Does Deep Active Learning Work in the Wild?

• URL: http://arxiv.org/abs/2302.00098v2
• Date: Fri, 06 Dec 2024 04:51:31 GMT
• Title: Does Deep Active Learning Work in the Wild?
• Authors: Simiao Ren, Saad Lahrichi, Yang Deng, Willie J. Padilla, Leslie Collins, Jordan Malof,
• Abstract summary: Deep active learning (DAL) methods have shown significant improvements in sample efficiency compared to simple random sampling.<n>Here, we argue that in real-world settings, or in the wild, there is significant uncertainty regarding good HPs.<n>We evaluate the performance of eleven modern DAL methods on eight benchmark problems.
• Score: 9.722499619824442
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Deep active learning (DAL) methods have shown significant improvements in sample efficiency compared to simple random sampling. While these studies are valuable, they nearly always assume that optimal DAL hyperparameter (HP) settings are known in advance, or optimize the HPs through repeating DAL several times with different HP settings. Here, we argue that in real-world settings, or in the wild, there is significant uncertainty regarding good HPs, and their optimization contradicts the premise of using DAL (i.e., we require labeling efficiency). In this study, we evaluate the performance of eleven modern DAL methods on eight benchmark problems as we vary a key HP shared by all methods: the pool ratio. Despite adjusting only one HP, our results indicate that eight of the eleven DAL methods sometimes underperform relative to simple random sampling and some frequently perform worse. Only three methods always outperform random sampling (albeit narrowly), and we find that these methods all utilize diversity to select samples - a relatively simple criterion. Our findings reveal the limitations of existing DAL methods when deployed in the wild, and present this as an important new open problem in the field.

Related papers

• Coupling Machine Learning Local Predictions with a Computational Fluid Dynamics Solver to Accelerate Transient Buoyant Plume Simulations [0.0]
  This study presents a versatile and scalable hybrid methodology, combining CFD and machine learning. The objective was to leverage local features to predict the temporal changes in the pressure field in comparable scenarios. Pressure estimates were employed as initial values to accelerate the pressure-velocity coupling procedure.
  arXiv  Detail & Related papers  (2024-09-11T10:38:30Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Deep Learning-Based Out-of-distribution Source Code Data Identification: How Far Have We Gone? [23.962076093344166]
  We propose an innovative deep learning-based approach addressing the OOD source code data identification problem. Our method is derived from an information-theoretic perspective with the use of innovative cluster-contrastive learning. Our method achieves a significantly higher performance from around 15.27%, 7.39%, and 4.93% on the FPR, AUROC, and AUPR measures, respectively.
  arXiv  Detail & Related papers  (2024-04-09T02:52:55Z)
• PETScML: Second-order solvers for training regression problems in Scientific Machine Learning [0.22499166814992438]
  In recent years, we have witnessed the emergence of scientific machine learning as a data-driven tool for the analysis. We introduce a software built on top of the Portable and Extensible Toolkit for Scientific computation to bridge the gap between deep-learning software and conventional machine-learning techniques.
  arXiv  Detail & Related papers  (2024-03-18T18:59:42Z)
• Autoencoder-based Anomaly Detection in Streaming Data with Incremental Learning and Concept Drift Adaptation [10.41066461952124]
  The paper proposes an autoencoder-based incremental learning method with drift detection (strAEm++DD) Our proposed method strAEm++DD leverages on the advantages of both incremental learning and drift detection. We conduct an experimental study using real-world and synthetic datasets with severe or extreme class imbalance, and provide an empirical analysis of strAEm++DD.
  arXiv  Detail & Related papers  (2023-05-15T19:40:04Z)
• Learning Controllable Adaptive Simulation for Multi-resolution Physics [86.8993558124143]
  We introduce Learning controllable Adaptive simulation for Multi-resolution Physics (LAMP) as the first full deep learning-based surrogate model. LAMP consists of a Graph Neural Network (GNN) for learning the forward evolution, and a GNN-based actor-critic for learning the policy of spatial refinement and coarsening. We demonstrate that our LAMP outperforms state-of-the-art deep learning surrogate models, and can adaptively trade-off computation to improve long-term prediction error.
  arXiv  Detail & Related papers  (2023-05-01T23:20:27Z)
• An Empirical Study on the Efficacy of Deep Active Learning for Image Classification [11.398892277968427]
  Deep Active Learning (DAL) has been advocated as a promising method to reduce labeling costs in supervised learning. Existing evaluations of DAL methods are based on different settings, and their results are controversial. This paper comprehensively evaluates 19 existing DAL methods in a uniform setting.
  arXiv  Detail & Related papers  (2022-11-30T17:44:59Z)
• Towards Robust Dataset Learning [90.2590325441068]
  We propose a principled, tri-level optimization to formulate the robust dataset learning problem. Under an abstraction model that characterizes robust vs. non-robust features, the proposed method provably learns a robust dataset.
  arXiv  Detail & Related papers  (2022-11-19T17:06:10Z)
• Towards Unsupervised HPO for Outlier Detection [23.77292404327994]
  We propose the first systematic approach called HPOD that is based on meta-learning. HPOD capitalizes on the prior performance of a large collection of HPs on existing OD benchmark datasets. It adapts (originally supervised) sequential model-based optimization to identify promising HPs efficiently.
  arXiv  Detail & Related papers  (2022-08-24T18:11:22Z)
• Hyperparameter Sensitivity in Deep Outlier Detection: Analysis and a Scalable Hyper-Ensemble Solution [21.130842136324528]
  We conduct the first large-scale analysis on the HP sensitivity of deep OD methods. We design a HP-robust and scalable deep hyper-ensemble model called ROBOD that assembles models with varying HP configurations.
  arXiv  Detail & Related papers  (2022-06-15T16:46:00Z)
• A Comparative Survey of Deep Active Learning [76.04825433362709]
  Active Learning (AL) is a set of techniques for reducing labeling cost by sequentially selecting data samples from a large unlabeled data pool for labeling. Deep Learning (DL) is data-hungry, and the performance of DL models scales monotonically with more training data. In recent years, Deep Active Learning (DAL) has risen as feasible solutions for maximizing model performance while minimizing the expensive labeling cost.
  arXiv  Detail & Related papers  (2022-03-25T05:17:24Z)
• A Lagrangian Duality Approach to Active Learning [119.36233726867992]
  We consider the batch active learning problem, where only a subset of the training data is labeled. We formulate the learning problem using constrained optimization, where each constraint bounds the performance of the model on labeled samples. We show, via numerical experiments, that our proposed approach performs similarly to or better than state-of-the-art active learning methods.
  arXiv  Detail & Related papers  (2022-02-08T19:18:49Z)
• Revisiting the Performance of iALS on Item Recommendation Benchmarks [19.704506591363256]
  Matrix factorization learned by implicit alternating least squares (iALS) is a popular baseline in recommender system research publications. Recent studies suggest that its prediction quality is not competitive with the current state of the art. We revisit four well-studied benchmarks where iALS was reported to perform poorly and show that with proper tuning, iALS is highly competitive.
  arXiv  Detail & Related papers  (2021-10-26T21:30:57Z)
• IQ-Learn: Inverse soft-Q Learning for Imitation [95.06031307730245]
  imitation learning from a small amount of expert data can be challenging in high-dimensional environments with complex dynamics. Behavioral cloning is a simple method that is widely used due to its simplicity of implementation and stable convergence. We introduce a method for dynamics-aware IL which avoids adversarial training by learning a single Q-function.
  arXiv  Detail & Related papers  (2021-06-23T03:43:10Z)
• Scalable Personalised Item Ranking through Parametric Density Estimation [53.44830012414444]
  Learning from implicit feedback is challenging because of the difficult nature of the one-class problem. Most conventional methods use a pairwise ranking approach and negative samplers to cope with the one-class problem. We propose a learning-to-rank approach, which achieves convergence speed comparable to the pointwise counterpart.
  arXiv  Detail & Related papers  (2021-05-11T03:38:16Z)
• Learning Sampling Policy for Faster Derivative Free Optimization [100.27518340593284]
  We propose a new reinforcement learning based ZO algorithm (ZO-RL) with learning the sampling policy for generating the perturbations in ZO optimization instead of using random sampling. Our results show that our ZO-RL algorithm can effectively reduce the variances of ZO gradient by learning a sampling policy, and converge faster than existing ZO algorithms in different scenarios.
  arXiv  Detail & Related papers  (2021-04-09T14:50:59Z)
• Bayesian Active Learning for Wearable Stress and Affect Detection [0.7106986689736827]
  Stress detection using on-device deep learning algorithms has been on the rise owing to advancements in pervasive computing. In this paper, we propose a framework with capabilities to represent model uncertainties through approximations in Bayesian Neural Networks. Our proposed framework achieves a considerable efficiency boost during inference, with a substantially low number of acquired pool points.
  arXiv  Detail & Related papers  (2020-12-04T16:19:37Z)
• Data-Efficient Learning for Complex and Real-Time Physical Problem Solving using Augmented Simulation [49.631034790080406]
  We present a task for navigating a marble to the center of a circular maze. We present a model that learns to move a marble in the complex environment within minutes of interacting with the real system.
  arXiv  Detail & Related papers  (2020-11-14T02:03:08Z)
• Provably Efficient Reward-Agnostic Navigation with Linear Value Iteration [143.43658264904863]
  We show how iteration under a more standard notion of low inherent Bellman error, typically employed in least-square value-style algorithms, can provide strong PAC guarantees on learning a near optimal value function. We present a computationally tractable algorithm for the reward-free setting and show how it can be used to learn a near optimal policy for any (linear) reward function.
  arXiv  Detail & Related papers  (2020-08-18T04:34:21Z)
• Practical and sample efficient zero-shot HPO [8.41866793161234]
  We provide an overview of available approaches and introduce two novel techniques to handle the problem. The first is based on a surrogate model and adaptively chooses pairs of dataset, configuration to query. The second is for settings where finding, tuning and testing a surrogate model is problematic, is a multi-fidelity technique combining HyperBand with submodular optimization.
  arXiv  Detail & Related papers  (2020-07-27T08:56:55Z)
• An Online Method for A Class of Distributionally Robust Optimization with Non-Convex Objectives [54.29001037565384]
  We propose a practical online method for solving a class of online distributionally robust optimization (DRO) problems. Our studies demonstrate important applications in machine learning for improving the robustness of networks.
  arXiv  Detail & Related papers  (2020-06-17T20:19:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.