Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests

• URL: http://arxiv.org/abs/2607.05457v1
• Date: Sun, 05 Jul 2026 22:07:22 GMT
• Title: Empirical Minimal-Realisation Compression of Deep Neural Networks via Controllability-Observability Tests
• Abstract summary: This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks.<n>By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians.
• Score: 10.32607023406232
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks often contain substantial hidden-state redundancy, but most compression methods operate directly on weights, neurons, or quantised representations without explicitly characterising the dynamical role of internal states. This paper proposes a controllability-observability framework for empirical state-order reduction of deep neural networks. By viewing a trained network as a depth-indexed nonlinear dynamical system, we construct data-driven reachability, observability, and balanced Gramians from hidden-state snapshots and output Jacobians. The resulting A/B/C tests estimate layer-wise reachable, observable, and jointly reachable--observable ranks. These ranks are then used not only as diagnostic measures of hidden-state redundancy, but also as actual compressed layer widths for realised reduced networks. Experiments on MNIST and CIFAR-10 compare the proposed balanced realisation against projection-based reduction, unstructured pruning, structured pruning, low-rank SVD, dynamic INT8 quantisation, and linear baselines. On MNIST, a four-layer SiLU DNN is reduced from state order 1024 to 277, giving 72.95% state compression and 73.48% parameter compression, while maintaining 95.45% accuracy compared with 96.60% for the full model. On CIFAR-10, a larger SiLU DNN is reduced from state order 4608 to 1339, giving 70.94% state compression and 83.09% parameter compression, while preserving accuracy from 54.45% to 54.44% and reducing CUDA inference latency by approximately 3X. The results show that balanced reachable-observable ranks provide a principled empirical minimal-realisation criterion for designing compact neural architectures with little or no loss in accuracy.

Related papers

• SparseKAN: Compressing Kolmogorov--Arnold Networks Across Basis Functions, Neurons, and Bits [6.860988566886594]
  We present textbfSparseKAN, a unified approach that compresses KANs along three complementary axes: basis functions, neurons/channels, and numerical precision.<n>SparseKAN equips the base branch, nonlinear basis branch, and individual basis terms with hierarchical learnable gates trained under a differentiable active-cost objective.
  arXiv  Detail & Related papers  (2026-08-01T20:42:41Z)
• PRISM: Sensitivity-Aware PolynoMial PRuning for EffIcient Neural Network Encryption [0.0]
  This paper presents a systematic reliability characterization of pruned CKKS-encritical neural networks.<n>Polynomial-Sensitivity-Aware Pruning (PSAP) is a structured pruning method that is inherently reliability-aware.
  arXiv  Detail & Related papers  (2026-07-20T05:47:06Z)
• Accelerated Volumetric Compression without Hierarchies: A Fourier Feature Based Implicit Neural Representation Approach [0.0]
  We introduce a structure-free neural compression method combining Fourierfeature encoding with selective voxel sampling.<n>Our dynamic voxel selection uses morphological dilation to prioritize active regions, reducing redundant computation without any hierarchical metadata.
  arXiv  Detail & Related papers  (2025-08-12T13:43:07Z)
• Reducing Storage of Pretrained Neural Networks by Rate-Constrained Quantization and Entropy Coding [56.066799081747845]
  The ever-growing size of neural networks poses serious challenges on resource-constrained devices.<n>We propose a novel post-training compression framework that combines rate-aware quantization with entropy coding.<n>Our method allows for very fast decoding and is compatible with arbitrary quantization grids.
  arXiv  Detail & Related papers  (2025-05-24T15:52:49Z)
• Edge AI: Evaluation of Model Compression Techniques for Convolutional Neural Networks [0.0]
  This work evaluates the compression techniques on ConvNeXt models in image classification tasks using the CIFAR-10 dataset. Results show significant reductions in model size, with up to 75% reduction achieved using structured pruning techniques. Dynamic quantization achieves a reduction of up to 95% in the number of parameters.
  arXiv  Detail & Related papers  (2024-09-02T11:48:19Z)
• "Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach [49.744093838327615]
  We provide a novel compression approach to wide and fully-connected emphdeep neural nets. Experiments on both synthetic and real-world data are conducted to support the advantages of the proposed compression scheme.
  arXiv  Detail & Related papers  (2024-03-01T03:46:28Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• Compact representations of convolutional neural networks via weight pruning and quantization [63.417651529192014]
  We propose a novel storage format for convolutional neural networks (CNNs) based on source coding and leveraging both weight pruning and quantization. We achieve a reduction of space occupancy up to 0.6% on fully connected layers and 5.44% on the whole network, while performing at least as competitive as the baseline.
  arXiv  Detail & Related papers  (2021-08-28T20:39:54Z)
• Spatio-Temporal Pruning and Quantization for Low-latency Spiking Neural Networks [6.011954485684313]
  Spiking Neural Networks (SNNs) are a promising alternative to traditional deep learning methods. However, a major drawback of SNNs is high inference latency. In this paper, we propose spatial and temporal pruning of SNNs.
  arXiv  Detail & Related papers  (2021-04-26T12:50:58Z)
• Toward Compact Deep Neural Networks via Energy-Aware Pruning [2.578242050187029]
  We propose a novel energy-aware pruning method that quantifies the importance of each filter in the network using nuclear-norm (NN) We achieve competitive results with 40.4/49.8% of FLOPs and 45.9/52.9% of parameter reduction with 94.13/94.61% in the Top-1 accuracy with ResNet-56/110 on CIFAR-10.
  arXiv  Detail & Related papers  (2021-03-19T15:33:16Z)
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems [77.88178159830905]
  Sparsity-Inducing Distribution-based Compression (SIDCo) is a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) Our evaluation shows SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.
  arXiv  Detail & Related papers  (2021-01-26T13:06:00Z)
• SCOP: Scientific Control for Reliable Neural Network Pruning [127.20073865874636]
  This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Redundant filters can be discovered in the adversarial process of different features. Our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet.
  arXiv  Detail & Related papers  (2020-10-21T03:02:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.