Effective Model Pruning

• URL: http://arxiv.org/abs/2509.25606v1
• Date: Tue, 30 Sep 2025 00:01:22 GMT
• Title: Effective Model Pruning
• Authors: Yixuan Wang, Dan Guralnik, Saiedeh Akbari, Warren Dixon,
• Abstract summary: We introduce Effective Model Pruning (EMP), a context-agnostic, parameter-free rule addressing a fundamental question about pruning: how many entries to keep.<n>EMP does not prescribe how to score the parameters or prune the models; instead, it supplies a universal adaptive threshold that can be applied to any pruning criterion.<n>We verify the effectiveness of N_eff by pruning the model with a scaled threshold beta*N_eff across a variety of criteria and models.
• Score: 3.8091555144548046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Effective Model Pruning (EMP), a context-agnostic, parameter-free rule addressing a fundamental question about pruning: how many entries to keep. EMP does not prescribe how to score the parameters or prune the models; instead, it supplies a universal adaptive threshold that can be applied to any pruning criterion: weight magnitude, attention score, KAN importance score, or even feature-level signals such as image pixel, and used on structural parts or weights of the models. Given any score vector s, EMP maps s to a built-in effective number N_eff which is inspired by the Inverse Simpson index of contributors. Retaining the N_eff highest scoring entries and zeroing the remainder yields sparse models with performance comparable to the original dense networks across MLPs, CNNs, Transformers/LLMs, and KAN, in our experiments. By leveraging the geometry of the simplex, we derive a tight lower bound on the preserved mass s_eff (the sum of retained scores) over the corresponding ordered probability simplex associated with the score vector s. We further verify the effectiveness of N_eff by pruning the model with a scaled threshold \b{eta}*N_eff across a variety of criteria and models. Experiments suggest that the default \b{eta} = 1 yields a robust threshold for model pruning while \b{eta} not equal to 1 still serves as an optional adjustment to meet specific sparsity requirements.

Related papers

• Model soups need only one ingredient [34.18140086731622]
  Fine-tuning large pre-trained models on a target distribution often improves in-distribution (ID) accuracy, but at the cost of robustness.<n>Weight-space ensembling methods, such as Model Soups, mitigate this effect by averaging multiple checkpoints.<n>We introduce MonoSoup, a simple, data-free, hyper parameter-free, post-hoc method that achieves a strong ID-OOD balance using only a single checkpoint.
  arXiv  Detail & Related papers  (2026-02-10T11:44:19Z)
• RANSAC Scoring Functions: Analysis and Reality Check [0.0]
  We revisit the problem of assigning a score (a quality of fit) to candidate geometric models.<n>We show that a threshold-based parameterization leads to a unified view of likelihood-based and robust M-estimators.
  arXiv  Detail & Related papers  (2025-12-22T20:08:46Z)
• pared: Model selection using multi-objective optimization [0.351124620232225]
  We present the R package pared to enable the use of multi-objective optimization for model selection.<n>Our approach entails the use of Gaussian process-based optimization to efficiently identify solutions that represent desirable trade-offs.
  arXiv  Detail & Related papers  (2025-05-27T20:20:04Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of Experts [64.34482582690927]
  We provide an efficient and principled method for sampling from a sequence of annealed, geometric-averaged, or product distributions derived from pretrained score-based models.<n>We propose Sequential Monte Carlo (SMC) resampling algorithms that leverage inference-time scaling to improve sampling quality.
  arXiv  Detail & Related papers  (2025-03-04T17:46:51Z)
• Partial sequence labeling with structured Gaussian Processes [8.239028141030621]
  We propose structured Gaussian Processes for partial sequence labeling. It encodes uncertainty in the prediction and does not need extra effort for model selection and hyper parameter learning. It is evaluated on several sequence labeling tasks and the experimental results show the effectiveness of the proposed model.
  arXiv  Detail & Related papers  (2022-09-20T00:56:49Z)
• Spike-and-Slab Generalized Additive Models and Scalable Algorithms for High-Dimensional Data [0.0]
  We propose hierarchical generalized additive models (GAMs) to accommodate high-dimensional data. We consider the smoothing penalty for proper shrinkage of curve and separation of smoothing function linear and nonlinear spaces. Two and deterministic algorithms, EM-Coordinate Descent and EM-Iterative Weighted Least Squares, are developed for different utilities.
  arXiv  Detail & Related papers  (2021-10-27T14:11:13Z)
• Probabilistic Circuits for Variational Inference in Discrete Graphical Models [101.28528515775842]
  Inference in discrete graphical models with variational methods is difficult. Many sampling-based methods have been proposed for estimating Evidence Lower Bound (ELBO) We propose a new approach that leverages the tractability of probabilistic circuit models, such as Sum Product Networks (SPN) We show that selective-SPNs are suitable as an expressive variational distribution, and prove that when the log-density of the target model is aweighted the corresponding ELBO can be computed analytically.
  arXiv  Detail & Related papers  (2020-10-22T05:04:38Z)
• Slice Sampling for General Completely Random Measures [74.24975039689893]
  We present a novel Markov chain Monte Carlo algorithm for posterior inference that adaptively sets the truncation level using auxiliary slice variables. The efficacy of the proposed algorithm is evaluated on several popular nonparametric models.
  arXiv  Detail & Related papers  (2020-06-24T17:53:53Z)
• SUMO: Unbiased Estimation of Log Marginal Probability for Latent Variable Models [80.22609163316459]
  We introduce an unbiased estimator of the log marginal likelihood and its gradients for latent variable models based on randomized truncation of infinite series. We show that models trained using our estimator give better test-set likelihoods than a standard importance-sampling based approach for the same average computational cost.
  arXiv  Detail & Related papers  (2020-04-01T11:49:30Z)
• On the Discrepancy between Density Estimation and Sequence Generation [92.70116082182076]
  log-likelihood is highly correlated with BLEU when we consider models within the same family. We observe no correlation between rankings of models across different families.
  arXiv  Detail & Related papers  (2020-02-17T20:13:35Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.