Neural expressiveness for beyond importance model compression

• URL: http://arxiv.org/abs/2512.06440v1
• Date: Sat, 06 Dec 2025 14:07:34 GMT
• Title: Neural expressiveness for beyond importance model compression
• Abstract summary: We introduce a novel criterion for model compression, named "Expressiveness"<n>Expressiveness emphasizes a neuron's or group of neurons ability to redistribute informational resources effectively.<n>We show that expressiveness is effectively approximated with arbitrary data or limited dataset's representative samples.
• Score: 2.536703706045127
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Neural Network Pruning has been established as driving force in the exploration of memory and energy efficient solutions with high throughput both during training and at test time. In this paper, we introduce a novel criterion for model compression, named "Expressiveness". Unlike existing pruning methods that rely on the inherent "Importance" of neurons' and filters' weights, ``Expressiveness" emphasizes a neuron's or group of neurons ability to redistribute informational resources effectively, based on the overlap of activations. This characteristic is strongly correlated to a network's initialization state, establishing criterion autonomy from the learning state stateless and thus setting a new fundamental basis for the expansion of compression strategies in regards to the "When to Prune" question. We show that expressiveness is effectively approximated with arbitrary data or limited dataset's representative samples, making ground for the exploration of Data-Agnostic strategies. Our work also facilitates a "hybrid" formulation of expressiveness and importance-based pruning strategies, illustrating their complementary benefits and delivering up to 10x extra gains w.r.t. weight-based approaches in parameter compression ratios, with an average of 1% in performance degradation. We also show that employing expressiveness (independently) for pruning leads to an improvement over top-performing and foundational methods in terms of compression efficiency. Finally, on YOLOv8, we achieve a 46.1% MACs reduction by removing 55.4\% of the parameters, with an increase of 3% in the mean Absolute Precision ($mAP_{50-95}$) for object detection on COCO dataset.

Related papers

• Compressing What Matters: Neuron Importance Meets Data-Aware Low Rank Approximation for Language Model Compression [39.47298454012977]
  We propose an enhanced and computationally efficient algorithm for dynamic compression rate allocation.<n> Experimental results support the efficacy of the proposed approach which performs on par or substantially better than the previous state-of-the-art.
  arXiv  Detail & Related papers  (2026-06-30T09:16:00Z)
• Selecting Samples on Graphs: A Unified Dataset Pruning Framework for Lossless Training Acceleration [71.77541419885925]
  We present a unified graph-based DP framework.<n>By modeling the dataset as a weighted graph, DP can be cast as a Maximum Weight Clique Problem (MWCP)<n>Our method outperforms existing DP methods while substantially reducing training cost, reducing training time by over 40% without sacrificing accuracy on ImageNet-1k with ResNet-50.
  arXiv  Detail & Related papers  (2026-06-11T05:13:32Z)
• Neural Network Compression by Approximate Differential Equivalence [0.9099663022952496]
  We propose an approach that compresses models by aggregating neurons rather than removing weights independently.<n>Our method encodes a trained network as induces a smooth ODE system and applies a lumping method called Approximate Forward Differential Equivalence.<n>We evaluate the method on synthetic datasets derived from nonlinear systems with known ground-truth behavior and on public regression benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T19:01:46Z)
• Explainability-Inspired Layer-Wise Pruning of Deep Neural Networks for Efficient Object Detection [3.317338104573978]
  We present an explainability-inspired, layer-wise pruning framework tailored for efficient object detection.<n>We conduct experiments across diverse object detection architectures, including ResNet-50, MobileNetV2, ShuffleNetV2, Faster R-CNN, RetinaNet, and YOLOv8.<n>Results show that the proposed attribution-inspired pruning consistently identifies different layers as least important compared to L1-norm-based methods.
  arXiv  Detail & Related papers  (2026-02-15T08:07:19Z)
• Knowledge-Informed Neural Network for Complex-Valued SAR Image Recognition [51.03674130115878]
  We introduce the Knowledge-Informed Neural Network (KINN), a lightweight framework built upon a novel "compression-aggregation-compression" architecture.<n>KINN establishes a state-of-the-art in parameter-efficient recognition, offering exceptional generalization in data-scarce and out-of-distribution scenarios.
  arXiv  Detail & Related papers  (2025-10-23T07:12:26Z)
• Linearity-based neural network compression [1.2200609701777907]
  We introduce the theory underlying this compression and evaluate our approach experimentally.<n>Applying our method on already importance-based pruned models shows very little interference between different types of compression.
  arXiv  Detail & Related papers  (2025-06-26T11:04:12Z)
• Stochastic Forward-Forward Learning through Representational Dimensionality Compression [14.020257767487323]
  We propose a novel goodness function dimensionality compression that uses the effective dimensionality of fluctuating neural responses to incorporate second-order statistical structure.<n>Our findings contribute to the development of more biologically plausible learning algorithms and suggest a natural fit for neuromorphic computing.
  arXiv  Detail & Related papers  (2025-05-22T13:19:29Z)
• Compressed Regression over Adaptive Networks [58.79251288443156]
  We derive the performance achievable by a network of distributed agents that solve, adaptively and in the presence of communication constraints, a regression problem. We devise an optimized allocation strategy where the parameters necessary for the optimization can be learned online by the agents.
  arXiv  Detail & Related papers  (2023-04-07T13:41:08Z)
• Modality-Agnostic Variational Compression of Implicit Neural Representations [96.35492043867104]
  We introduce a modality-agnostic neural compression algorithm based on a functional view of data and parameterised as an Implicit Neural Representation (INR) Bridging the gap between latent coding and sparsity, we obtain compact latent representations non-linearly mapped to a soft gating mechanism. After obtaining a dataset of such latent representations, we directly optimise the rate/distortion trade-off in a modality-agnostic space using neural compression.
  arXiv  Detail & Related papers  (2023-01-23T15:22:42Z)
• WeightMom: Learning Sparse Networks using Iterative Momentum-based pruning [0.0]
  We propose a weight based pruning approach in which the weights are pruned gradually based on their momentum of the previous iterations. We evaluate our approach on networks such as AlexNet, VGG16 and ResNet50 with image classification datasets such as CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-08-11T07:13:59Z)
• Compact representations of convolutional neural networks via weight pruning and quantization [63.417651529192014]
  We propose a novel storage format for convolutional neural networks (CNNs) based on source coding and leveraging both weight pruning and quantization. We achieve a reduction of space occupancy up to 0.6% on fully connected layers and 5.44% on the whole network, while performing at least as competitive as the baseline.
  arXiv  Detail & Related papers  (2021-08-28T20:39:54Z)
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems [77.88178159830905]
  Sparsity-Inducing Distribution-based Compression (SIDCo) is a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) Our evaluation shows SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.
  arXiv  Detail & Related papers  (2021-01-26T13:06:00Z)
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation [53.8171136907856]
  We introduce a set of simple yet effective data augmentation strategies dubbed cutoff. cutoff relies on sampling consistency and thus adds little computational overhead. cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.
  arXiv  Detail & Related papers  (2020-09-29T07:08:35Z)
• Data-Independent Structured Pruning of Neural Networks via Coresets [21.436706159840018]
  We propose the first efficient structured pruning algorithm with a provable trade-off between its compression rate and the approximation error for any future test sample. Unlike previous works, our coreset is data independent, meaning that it provably guarantees the accuracy of the function for any input $xin mathbbRd$, including an adversarial one.
  arXiv  Detail & Related papers  (2020-08-19T08:03:09Z)
• Exploiting Non-Linear Redundancy for Neural Model Compression [26.211513643079993]
  We propose a novel model compression approach based on exploitation of linear dependence. Our method results in a reduction of up to 99% in overall network size with small loss in performance.
  arXiv  Detail & Related papers  (2020-05-28T15:13:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.