One Loop, Two Gains: Can Active Learning win the Lottery for Free?

• URL: http://arxiv.org/abs/2609.10311v1
• Date: Wed, 09 Sep 2026 15:18:37 GMT
• Title: One Loop, Two Gains: Can Active Learning win the Lottery for Free?
• Abstract summary: We show that I&P can integrate magnitude pruning into each active learning retraining cycle at practically no additional cost.<n>Our results demonstrate that I&P yields sparse, deployable models at each active learning iteration.<n>These per-it sparse models can address two computational bottlenecks - per-round model retraining and acquisition scoring over the unlabeled pool.
• Score: 2.181071341517602
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The lottery ticket hypothesis posits the existence of winning tickets: sparse subnetworks that, when trained in isolation from their original initialization, match the accuracy of the full dense network. The predominant method for discovering such tickets, iterative magnitude pruning, alternates pruning with full retraining from scratch until convergence over many cycles. Similarly, deep active learning also retrains a model from scratch after each acquisition round as new labels become available. Despite this shared reliance on iterative retraining with a substantial computational overhead, the two paradigms have been studied separately. We observe that the iterative training loop inherent to pool-based active learning already provides the exact computational structure that iterative magnitude pruning exploits, and propose Improve & Prune (I&P), a method that integrates magnitude pruning into each active learning retraining cycle at practically no additional cost. This raises a key empirical question: can iterative magnitude pruning produce winning tickets under the non-stationary data regime of active learning? We investigate this question across multiple acquisition functions, architecture families, and image classification datasets, including an active fine-tuning scenario. Our results demonstrate that I&P yields sparse, deployable models at each active learning iteration. Those match the accuracy of their dense counterparts at sparsities up to 95%, effectively obtaining winning tickets as a byproduct of the active learning pipeline. These per-iteration sparse models can address two computational bottlenecks - per-round model retraining and acquisition scoring over the unlabeled pool - that currently prevent the practical adoption of DAL on large architectures and large unlabeled pools.

Related papers

• Playing the Lottery With Concave Regularizers for Sparse Trainable Neural Networks [10.48836159692231]
  We propose a novel class of methods to play the lottery.<n>The key point is the use of concave regularization to promote the sparsity of a relaxed binary mask.<n>We show that the proposed method can improve the performance of state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2025-01-19T18:05:13Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• BatchGFN: Generative Flow Networks for Batch Active Learning [80.73649229919454]
  BatchGFN is a novel approach for pool-based active learning that uses generative flow networks to sample sets of data points proportional to a batch reward. We show our approach enables principled sampling near-optimal utility batches at inference time with a single forward pass per point in the batch in toy regression problems.
  arXiv  Detail & Related papers  (2023-06-26T20:41:36Z)
• Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models [106.19385911520652]
  Lottery Ticket Hypothesis (LTH) and its variants have been exploited to prune large pre-trained models generating parameterworks. LTH is enormously inhibited by repetitive full training and pruning routine of iterative magnitude pruning (IMP) We propose Instant Soup Pruning (ISP) to generate lottery ticket quality IMPworks.
  arXiv  Detail & Related papers  (2023-06-18T03:09:52Z)
• LOFT: Finding Lottery Tickets through Filter-wise Training [15.06694204377327]
  We show how one can efficiently identify the emergence of such winning tickets, and use this observation to design efficient pretraining algorithms. We present the emphLOttery ticket through Filter-wise Training algorithm, dubbed as textscLoFT. Experiments show that textscLoFT $i)$ preserves and finds good lottery tickets, while $ii)$ achieves it non-trivial and communication savings.
  arXiv  Detail & Related papers  (2022-10-28T14:43:42Z)
• Dual Lottery Ticket Hypothesis [71.95937879869334]
  Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark. We propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH.
  arXiv  Detail & Related papers  (2022-03-08T18:06:26Z)
• Accelerating Deep Learning with Dynamic Data Pruning [0.0]
  Deep learning has become prohibitively costly, requiring access to powerful computing systems to train state-of-the-art networks. Previous work, such as forget scores and GraNd/EL2N scores, identify important samples within a full dataset and pruning the remaining samples, thereby reducing the iterations per epoch. We propose two algorithms, based on reinforcement learning techniques, to dynamically prune samples and achieve even higher accuracy than the random dynamic method.
  arXiv  Detail & Related papers  (2021-11-24T16:47:34Z)
• FreeTickets: Accurate, Robust and Efficient Deep Ensemble by Training with Dynamic Sparsity [74.58777701536668]
  We introduce the FreeTickets concept, which can boost the performance of sparse convolutional neural networks over their dense network equivalents by a large margin. We propose two novel efficient ensemble methods with dynamic sparsity, which yield in one shot many diverse and accurate tickets "for free" during the sparse training process.
  arXiv  Detail & Related papers  (2021-06-28T10:48:20Z)
• The Elastic Lottery Ticket Hypothesis [106.79387235014379]
  Lottery Ticket Hypothesis raises keen attention to identifying sparse trainableworks or winning tickets. The most effective method to identify such winning tickets is still Iterative Magnitude-based Pruning. We propose a variety of strategies to tweak the winning tickets found from different networks of the same model family.
  arXiv  Detail & Related papers  (2021-03-30T17:53:45Z)
• Lottery Ticket Implies Accuracy Degradation, Is It a Desirable Phenomenon? [43.47794674403988]
  In deep model compression, the recent finding "Lottery Ticket Hypothesis" (LTH) (Frankle & Carbin) pointed out that there could exist a winning ticket. We investigate the underlying condition and rationale behind the winning property, and find that the underlying reason is largely attributed to the correlation between weights and final-trained weights. We propose the "pruning & fine-tuning" method that consistently outperforms lottery ticket sparse training.
  arXiv  Detail & Related papers  (2021-02-19T14:49:46Z)
• Good Students Play Big Lottery Better [84.6111281091602]
  Lottery ticket hypothesis suggests that a dense neural network contains a sparse sub-network that can match the test accuracy of the original dense net. Recent studies demonstrate that a sparse sub-network can still be obtained by using a rewinding technique. This paper proposes a new, simpler and yet powerful technique for re-training the sub-network, called "Knowledge Distillation ticket" (KD ticket)
  arXiv  Detail & Related papers  (2021-01-08T23:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.