Diluting Restricted Boltzmann Machines

• URL: http://arxiv.org/abs/2511.00648v1
• Date: Sat, 01 Nov 2025 18:03:58 GMT
• Title: Diluting Restricted Boltzmann Machines
• Authors: C. Díaz-Faloh, R. Mulet,
• Abstract summary: This paper investigates whether simpler, sparser networks can maintain strong performance under extreme pruning conditions.<n>Inspired by the Lottery Ticket Hypothesis, we demonstrate that RBMs can achieve high-quality generative performance even when up to 80% of the connections are pruned before training.<n>We identify a sharp transition above which the generative quality degrades abruptly when pruning disrupts a minimal core of essential connections.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in artificial intelligence have relied heavily on increasingly large neural networks, raising concerns about their computational and environmental costs. This paper investigates whether simpler, sparser networks can maintain strong performance by studying Restricted Boltzmann Machines (RBMs) under extreme pruning conditions. Inspired by the Lottery Ticket Hypothesis, we demonstrate that RBMs can achieve high-quality generative performance even when up to 80% of the connections are pruned before training, confirming that they contain viable sub-networks. However, our experiments reveal crucial limitations: trained networks cannot fully recover lost performance through retraining once additional pruning is applied. We identify a sharp transition above which the generative quality degrades abruptly when pruning disrupts a minimal core of essential connections. Moreover, re-trained networks remain constrained by the parameters originally learned performing worse than networks trained from scratch at equivalent sparsity levels. These results suggest that for sparse networks to work effectively, pruning should be implemented early in training rather than attempted afterwards. Our findings provide practical insights for the development of efficient neural architectures and highlight the persistent influence of initial conditions on network capabilities.

Related papers

• Network Sparsity Unlocks the Scaling Potential of Deep Reinforcement Learning [57.3885832382455]
  We show that introducing static network sparsity alone can unlock further scaling potential beyond dense counterparts with state-of-the-art architectures.<n>Our analysis reveals that, in contrast to naively scaling up dense DRL networks, such sparse networks achieve both higher parameter efficiency for network expressivity.
  arXiv  Detail & Related papers  (2025-06-20T17:54:24Z)
• SynA-ResNet: Spike-driven ResNet Achieved through OR Residual Connection [10.702093960098104]
  Spiking Neural Networks (SNNs) have garnered substantial attention in brain-like computing for their biological fidelity and the capacity to execute energy-efficient spike-driven operations. We propose a novel training paradigm that first accumulates a large amount of redundant information through OR Residual Connection (ORRC) We then filters out the redundant information using the Synergistic Attention (SynA) module, which promotes feature extraction in the backbone while suppressing the influence of noise and useless features in the shortcuts.
  arXiv  Detail & Related papers  (2023-11-11T13:36:27Z)
• Stimulative Training++: Go Beyond The Performance Limits of Residual Networks [91.5381301894899]
  Residual networks have shown great success and become indispensable in recent deep neural network models. Previous research has suggested that residual networks can be considered as ensembles of shallow networks. We identify a problem that is analogous to social loafing, whereworks within a residual network are prone to exert less effort when working as part of a group compared to working alone.
  arXiv  Detail & Related papers  (2023-05-04T02:38:11Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training [111.15069968583042]
  Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. We empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent. Our results strongly suggest there is larger-than-expected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning.
  arXiv  Detail & Related papers  (2022-02-05T21:19:41Z)
• Back to Basics: Efficient Network Compression via IMP [22.586474627159287]
  Iterative Magnitude Pruning (IMP) is one of the most established approaches for network pruning. IMP is often argued that it reaches suboptimal states by not incorporating sparsification into the training phase. We find that IMP with SLR for retraining can outperform state-of-the-art pruning-during-training approaches.
  arXiv  Detail & Related papers  (2021-11-01T11:23:44Z)
• How much pre-training is enough to discover a good subnetwork? [10.699603774240853]
  We mathematically analyze the amount of dense network pre-training needed for a pruned network to perform well. We find a simple theoretical bound in the number of gradient descent pre-training iterations on a two-layer, fully-connected network. Experiments with larger datasets require more pre-training forworks obtained via pruning to perform well.
  arXiv  Detail & Related papers  (2021-07-31T15:08:36Z)
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning [83.59005356327103]
  A common limitation of most existing pruning techniques is that they require pre-training of the network at least once before pruning. We propose STAMP, which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. We validate STAMP against recent advanced pruning methods on benchmark datasets.
  arXiv  Detail & Related papers  (2020-06-22T10:57:43Z)
• Robust Pruning at Initialization [61.30574156442608]
  A growing need for smaller, energy-efficient, neural networks to be able to use machine learning applications on devices with limited computational resources. For Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned.
  arXiv  Detail & Related papers  (2020-02-19T17:09:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.