Optimal Pruning for Neural Architectures using Fisher Information Distances

• URL: http://arxiv.org/abs/2609.16129v1
• Date: Mon, 14 Sep 2026 18:00:03 GMT
• Title: Optimal Pruning for Neural Architectures using Fisher Information Distances
• Abstract summary: Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes.<n>The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space.<n>By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined.
• Score: 0.15554979526414522
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A new scheme for parameter pruning is introduced, derived from the differential-geometric distance in model space. Pruning a parameter sets its value to zero, representing a displacement of the model to the hypersurface on which that parameter vanishes. The minimal distance from the unpruned model to this hypersurface is naturally computed via the geodesic distance in the model space as determined by the Fisher information metric. This distance determines the true change in the model, and its performance, under pruning. By analysing progressively more faithful approximations of this geodesic distance a natural hierarchy of optimality for pruning methods is determined. This starts with the traditional magnitude pruning, then develops into new more sophisticated and effective pruning schemes. The method is demonstrated for both fully-connected networks and vision transformers, on MNIST and CIFAR-10, over the complete $0$-$100\%$ pruning range and across five random seeds. It outperforms pruning by parameter magnitude and by the local Fisher information alone in every architecture and dataset combination considered, on both accuracy and the Matthews correlation coefficient. Additionally, analysis of different levels of geodesic approximation produces intermediate pruning schemes that are computationally efficient and maintain near-optimal performance. This geometric picture supplies not only a state-of-the-art pruning methodology for AI models, but also a verified and mathematically-motivated justification for pruning schemes.

Related papers

• A Defense of the Quadratic Model [71.57026347416729]
  We stress test the simplest possible model of optimization -- the quadratic model.<n>We show that it can be predictive surprisingly in an LLM setting with 150M parameters and 3B training tokens.
  arXiv  Detail & Related papers  (2026-07-23T18:02:03Z)
• Tubular Riemannian Laplace Approximations for Bayesian Neural Networks [0.0]
  Laplace approximations are among the simplest and most practical methods for approximate Bayesian inference in neural networks.<n>Recent work has proposed geometric Gaussian approximations to adapt to this structure.<n>We introduce the Tubular Riemannian Laplace (TRL) approximation.
  arXiv  Detail & Related papers  (2025-12-30T17:50:55Z)
• Lotus-2: Advancing Geometric Dense Prediction with Powerful Image Generative Model [32.831576387973875]
  We propose a two-stage deterministic framework for stable, accurate and fine-grained geometric dense prediction.<n>Specifically, in the first stage, the core predictor employs a single-step deterministic formulation with a clean-data objective.<n>In the second stage, the detail sharpener performs a constrained multi-step rectified-flow refinement within the manifold defined by the core predictor.
  arXiv  Detail & Related papers  (2025-11-30T18:57:25Z)
• An End-to-End Differentiable, Graph Neural Network-Embedded Pore Network Model for Permeability Prediction [0.42970700836450487]
  Pore network models (PNMs) estimate pore-scale hydraulic conductance, limiting their accuracy in complex structures.<n>We present an end-to-end differentiable hybrid framework that embeds a graph neural network (GNN) into a PNM.<n>The model achieves high accuracy and adjoins well across different scales, outperforming both pure data-driven and traditional PNM approaches.
  arXiv  Detail & Related papers  (2025-09-17T09:15:23Z)
• Geometric Operator Learning with Optimal Transport [77.16909146519227]
  We propose integrating optimal transport (OT) into operator learning for partial differential equations (PDEs) on complex geometries.<n>For 3D simulations focused on surfaces, our OT-based neural operator embeds the surface geometry into a 2D parameterized latent space.<n> Experiments with Reynolds-averaged Navier-Stokes equations (RANS) on the ShapeNet-Car and DrivAerNet-Car datasets show that our method achieves better accuracy and also reduces computational expenses.
  arXiv  Detail & Related papers  (2025-07-26T21:28:25Z)
• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels [78.6096486885658]
  We introduce lower bounds to the linearized Laplace approximation of the marginal likelihood. These bounds are amenable togradient-based optimization and allow to trade off estimation accuracy against computational complexity.
  arXiv  Detail & Related papers  (2023-06-06T19:02:57Z)
• Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees [57.67528738886731]
  We study the numerical stability of scalable sparse approximations based on inducing points. For low-dimensional tasks such as geospatial modeling, we propose an automated method for computing inducing points satisfying these conditions.
  arXiv  Detail & Related papers  (2022-10-14T15:20:17Z)
• Structured Optimal Variational Inference for Dynamic Latent Space Models [16.531262817315696]
  We consider a latent space model for dynamic networks, where our objective is to estimate the pairwise inner products plus the intercept of the latent positions. To balance posterior inference and computational scalability, we consider a structured mean-field variational inference framework.
  arXiv  Detail & Related papers  (2022-09-29T22:10:42Z)
• Scaling Structured Inference with Randomization [64.18063627155128]
  We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states. Our method is widely applicable to classical DP-based inference. It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
  arXiv  Detail & Related papers  (2021-12-07T11:26:41Z)
• Inverting brain grey matter models with likelihood-free inference: a tool for trustable cytoarchitecture measurements [62.997667081978825]
  characterisation of the brain grey matter cytoarchitecture with quantitative sensitivity to soma density and volume remains an unsolved challenge in dMRI. We propose a new forward model, specifically a new system of equations, requiring a few relatively sparse b-shells. We then apply modern tools from Bayesian analysis known as likelihood-free inference (LFI) to invert our proposed model.
  arXiv  Detail & Related papers  (2021-11-15T09:08:27Z)
• An Artificial Intelligence System for Combined Fruit Detection and Georeferencing, Using RTK-Based Perspective Projection in Drone Imagery [0.0]
  This work presents an Artificial Intelligence (AI) system, which detects and counts apples from aerial drone imagery of commercial orchards. To reduce computational cost, a novel precursory stage to the network is designed to preprocess raw imagery into cropped images of individual trees. Unique geospatial identifiers are allocated to these using the perspective projection model. Experiments show that a k-means clustering approach, never before seen in literature for Faster R-CNN, resulted in the most significant improvements to calibrated mAP.
  arXiv  Detail & Related papers  (2021-01-01T23:39:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.