Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates

• URL: http://arxiv.org/abs/2603.08914v1
• Date: Mon, 09 Mar 2026 20:33:16 GMT
• Title: Uncovering a Winning Lottery Ticket with Continuously Relaxed Bernoulli Gates
• Abstract summary: Strong Lottery Ticket (SLT) hypothesis suggests randomly networks contain sparseworks achieving competitive accuracy without weight training.<n>We propose using continuously relaxed Bernoulligating to discover SLTs through fully differentiable, end-to-end optimization.<n> Experiments across fully connected networks, CNNs (ResNet, Wide-ResNet), and Vision Transformers (ViT, Swin-T) demonstrate up to 90% sparsity with minimal accuracy loss.
• Score: 13.591018807414484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Over-parameterized neural networks incur prohibitive memory and computational costs for resource-constrained deployment. The Strong Lottery Ticket (SLT) hypothesis suggests that randomly initialized networks contain sparse subnetworks achieving competitive accuracy without weight training. Existing SLT methods, notably edge-popup, rely on non-differentiable score-based selection, limiting optimization efficiency and scalability. We propose using continuously relaxed Bernoulli gates to discover SLTs through fully differentiable, end-to-end optimization - training only gating parameters while keeping all network weights frozen at their initialized values. Continuous relaxation enables direct gradient-based optimization of an $\ell_0$-regularization objective, eliminating the need for non-differentiable gradient estimators or iterative pruning cycles. To our knowledge, this is the first fully differentiable approach for SLT discovery that avoids straight-through estimator approximations. Experiments across fully connected networks, CNNs (ResNet, Wide-ResNet), and Vision Transformers (ViT, Swin-T) demonstrate up to 90% sparsity with minimal accuracy loss - nearly double the sparsity achieved by edge-popup at comparable accuracy - establishing a scalable framework for pre-training network sparsification.

Related papers

• Double-Scoring: Reliable Extraction of Strong Lottery Tickets [0.0]
  Lottery ticket hypothesis proposes that large random neural networks contain sparseworks that can match the performance of dense models after comparable training.<n>Existing theory establishes that such strong lottery tickets exist, but reliable extraction remains difficult.<n>We revisit edge-popup, a frozen-weight score-training method for extracting strong tickets, and identify layerwise sparsity selection as a central bottleneck.<n>We show that fixed-density masking in an augmented score space preserves access to all original-coordinate masks, and we show that the resulting method can be interpreted as edge-popup on a zero-augmented network.
  arXiv  Detail & Related papers  (2026-07-15T20:20:43Z)
• Direct Bethe Free Energy Minimization for Bayesian Neural Network [0.0]
  We propose training Bayesian neural networks by directly minimizing the Bethe free energy rather than maximizing a variational lower bound.<n>On tree-structured factor graphs the Bethe free energy is exact; deterministic layers drop out of the objective and are trained by standard backpropagation.<n>All variants admit a closed-form predictive at MAP-equivalent inference cost, in contrast to ensemble and sampling-based methods.
  arXiv  Detail & Related papers  (2026-05-08T20:12:48Z)
• Winning the Lottery by Preserving Network Training Dynamics with Concrete Ticket Search [0.5156484100374058]
  Lottery Ticket Hypothesis asserts existence of highly sparse, trainableworks ('winning tickets') within dense, randomly neural networks.<n>State-of-the-art methods of drawing these tickets, like Lottery Ticket Rewinding, are computationally prohibitive.<n>In this work, we argue that PaI's reliance on first-order saliency metrics, which ignore inter-weight dependencies, contributes substantially to this performance gap.
  arXiv  Detail & Related papers  (2025-12-08T03:48:51Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Preserving Deep Representations In One-Shot Pruning: A Hessian-Free Second-Order Optimization Framework [12.331056472174275]
  We present SNOWS, a one-shot post-training pruning framework aimed at reducing the cost of vision network inference without retraining.<n>A key innovation of our framework is the use of Hessian-free optimization to compute exact Newton descent steps without needing to compute or store the full Hessian matrix.
  arXiv  Detail & Related papers  (2024-11-27T14:25:00Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Learning a Consensus Sub-Network with Polarization Regularization and One Pass Training [2.895034191799291]
  Pruning schemes create extra overhead either by iterative training and fine-tuning for static pruning or repeated computation of a dynamic pruning graph.<n>We propose a new parameter pruning strategy for learning a lighter-weight sub-network that minimizes the energy cost while maintaining comparable performance to the fully parameterised network on given downstream tasks.<n>Our results on CIFAR-10, CIFAR-100, and Tiny Imagenet suggest that our scheme can remove 50% of connections in deep networks with 1% reduction in classification accuracy.
  arXiv  Detail & Related papers  (2023-02-17T09:37:17Z)
• Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach [64.23331120621118]
  This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks. We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network. We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
  arXiv  Detail & Related papers  (2022-08-08T03:13:24Z)
• FreeTickets: Accurate, Robust and Efficient Deep Ensemble by Training with Dynamic Sparsity [74.58777701536668]
  We introduce the FreeTickets concept, which can boost the performance of sparse convolutional neural networks over their dense network equivalents by a large margin. We propose two novel efficient ensemble methods with dynamic sparsity, which yield in one shot many diverse and accurate tickets "for free" during the sparse training process.
  arXiv  Detail & Related papers  (2021-06-28T10:48:20Z)
• Enabling certification of verification-agnostic networks via memory-efficient semidefinite programming [97.40955121478716]
  We propose a first-order dual SDP algorithm that requires memory only linear in the total number of network activations. We significantly improve L-inf verified robust accuracy from 1% to 88% and 6% to 40% respectively. We also demonstrate tight verification of a quadratic stability specification for the decoder of a variational autoencoder.
  arXiv  Detail & Related papers  (2020-10-22T12:32:29Z)
• Progressive Skeletonization: Trimming more fat from a network at initialization [76.11947969140608]
  We propose an objective to find a skeletonized network with maximum connection sensitivity. We then propose two approximate procedures to maximize our objective. Our approach provides remarkably improved performance on higher pruning levels.
  arXiv  Detail & Related papers  (2020-06-16T11:32:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.