Denoised Variance-Based Pruning with Optimal Brain Bias Compensation

• URL: http://arxiv.org/abs/2608.17657v1
• Date: Tue, 18 Aug 2026 11:23:49 GMT
• Title: Denoised Variance-Based Pruning with Optimal Brain Bias Compensation
• Abstract summary: Variance-Based Pruning (VBP) introduced a promising paradigm by selecting neurons based on activation variance.<n>We introduce Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB$2$C)<n>It achieves state-of-the-art training-free performance; at 50% pruning, it retains over 90% of the original Top-1 accuracy.
• Score: 45.88028371034407
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision Transformers (ViTs) achieve state-of-the-art performance but carry massive computational overhead that restricts edge deployment. Although structural pruning has emerged as a key strategy to reduce these costs, existing methods often suffer from severe accuracy degradation or require expensive retraining. Recently, Variance-Based Pruning (VBP) introduced a promising paradigm by selecting neurons based on activation variance; however, it remains limited by statistical noise in finite-sample activation covariance and reliance on bias-only updates that cannot fully account for structural reconstruction error. To address these limitations, we introduce Denoised Variance-Based Pruning with Optimal Brain Bias Compensation (DVBP + OB$^2$C). We leverage random matrix theory to filter noise from the activation covariance spectrum for robust neuron selection and mathematically prove that integrating mean-shift compensation into the Optimal Brain Compression objective reduces the layer-wise Hessian exactly to the activation covariance matrix. This enables an optimal, closed-form update of the remaining weights using the same statistics gathered for selection. Extensive experiments on DeiT, Swin, and ConvNeXt architectures demonstrate that DVBP + OB$^2$C achieves state-of-the-art training-free performance; at 50% MLP pruning, it retains over 90% of the original Top-1 accuracy on Small and Base variants, outperforming VBP by up to 29.46% (ConvNeXt-T) and 7.33% (Swin-S). The code is available at: https://github.com/geontackee/DVBP_OB2C.

Related papers

• Error-Conditioned Neural Solvers [36.51479208528461]
  Neural surrogate models offer fast approximate mappings from PDE parameters to solutions, but they typically treat solving as a purely statistical task.<n>Recent hybrid methods promote physical correctness by targeting the PDE residual via gradient descent or Gauss-Newton steps, but inherit the compute cost and instability of the underlying classical gradients.<n>We show, theoretically and empirically, that numerically minimizing the PDE residual can be an unreliable proxy for reconstruction accuracy in ill-conditioned systems.
  arXiv  Detail & Related papers  (2026-06-25T17:56:27Z)
• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration [71.77541419885925]
  We present a unified graph-based DP framework.<n>By modeling the dataset as a weighted graph, DP can be cast as a Maximum Weight Clique Problem (MWCP)<n>Our method outperforms existing DP methods while substantially reducing training cost, reducing training time by over 40% without sacrificing accuracy on ImageNet-1k with ResNet-50.
  arXiv  Detail & Related papers  (2026-06-11T05:13:32Z)
• Residual-Controlled Multiplier Learning for Stochastic Constrained Decision-Making [12.04727482907937]
  CML addresses the problem of tracking the noise of primal feedback multipliers under mini-batch conditions.<n>We decompose the multiplier into an effective pressure signal for primal descent and a finite-gain multiplier.
  arXiv  Detail & Related papers  (2026-06-05T09:35:22Z)
• Direct Bethe Free Energy Minimization for Bayesian Neural Network [0.0]
  We propose training Bayesian neural networks by directly minimizing the Bethe free energy rather than maximizing a variational lower bound.<n>On tree-structured factor graphs the Bethe free energy is exact; deterministic layers drop out of the objective and are trained by standard backpropagation.<n>All variants admit a closed-form predictive at MAP-equivalent inference cost, in contrast to ensemble and sampling-based methods.
  arXiv  Detail & Related papers  (2026-05-08T20:12:48Z)
• DARTS: Targeting Prognostic Covariates in Budget-Constrained Sequential Experiments [47.61857875238484]
  We introduce Dynamic Adaptive Rerandomization via Thompson Sampling (DARTS)<n>DARTS treats covariate acquisition as a sequential optimization problem embedded within a design-based causal inference task.<n>We derive a Bayes risk bound for the acquisition layer that matches the minimax lower bound up to logarithmic factors.
  arXiv  Detail & Related papers  (2026-05-07T17:27:51Z)
• ORTHOBO: Orthogonal Bayesian Hyperparameter Optimization [31.941089890706348]
  noisy acquisition estimates can lead to unstable decisions.<n>We introduce OrthoBO: a Bayesian optimization framework with ensemble surrogates and an outer log transformation.<n>We show theoretically that our estimator preserves the target, leads to variance reduction, and improves pairwise ranking stability.
  arXiv  Detail & Related papers  (2026-05-07T15:49:03Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• From Noisy Traces to Stable Gradients: Bias-Variance Optimized Preference Optimization for Aligning Large Reasoning Models [90.45197506653341]
  Large reasoning models generate intermediate reasoning traces before producing final answers.<n> aligning LRMs with human preferences, a crucial prerequisite for model deployment, remains underexplored.<n>A common workaround optimized a single sampled trajectory, which introduces substantial gradient variance from trace sampling.
  arXiv  Detail & Related papers  (2025-10-06T17:58:01Z)
• NIRVANA: Structured pruning reimagined for large language models compression [50.651730342011014]
  We introduce NIRVANA, a novel pruning method designed to balance immediate zero-shot preservation accuracy with robust fine-tuning.<n>To further address the unique challenges posed by structured pruning, NIRVANA incorporates an adaptive sparsity allocation mechanism across layers and modules.<n>Experiments conducted on Llama3, Qwen, T5 models demonstrate that NIRVANA outperforms existing structured pruning methods under equivalent sparsity constraints.
  arXiv  Detail & Related papers  (2025-09-17T17:59:00Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.