From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy

• URL: http://arxiv.org/abs/2609.04881v1
• Date: Fri, 04 Sep 2026 08:38:35 GMT
• Title: From Deep to Shallow: Unconstrained and Efficient Layer Merging Strategy
• Abstract summary: Deep Neural Networks have become foundational in many areas of Machine Learning.<n>Deep compression methods have been proposed to identify and linearize redundant activation functions.<n>These methods face two key challenges: they cannot be directly applied to convolutions with padding, and they typically increase the kernel size of merged layers.<n>We propose an efficient strategy that enables merging of layers without an existing analytical solution, and also without increasing kernel size.
• Score: 13.583841141384008
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Deep Neural Networks have become foundational in many areas of Machine Learning, high computational demands limit their application in resource-constrained environments. To address this issue, depth compression methods have been proposed to identify and linearize redundant activation functions, thereby allowing for the merging of layers without intermediate non-linearities. However, these methods face two key challenges: they cannot be directly applied to convolutions with padding due to the absence of an analytical solution for merging these layers, and they typically increase the kernel size of merged layers, thus limiting speed-up gains. To overcome these limitations, we propose an efficient strategy that enables merging of layers without an existing analytical solution, and also without increasing kernel size. We validate our approach across multiple architectures and datasets, and measure inference speed-up gains on real embedded platforms. We publicly released the code at https://github.com/ShulzhenkoPetr/deep-to-shallow.

Related papers

• Towards Practical Lossless Neural Compression for LiDAR Point Clouds [84.36825469211375]
  extreme sparsity of high-precision geometric details hinders efficient context modeling.<n>Our framework comprises two lightweight modules.<n>Experiments demonstrate competitive compression performance at real-time speed.
  arXiv  Detail & Related papers  (2026-03-26T10:02:07Z)
• Geminet: Learning the Duality-based Iterative Process for Lightweight Traffic Engineering in Changing Topologies [53.38648279089736]
  Geminet is a lightweight and scalable ML-based TE framework that can handle changing topologies.<n>Its neural network size is only 0.04% to 7% of existing schemes.<n>When trained on large-scale topologies, Geminet consumes under 10 GiB of memory, more than eight times less than the 80-plus GiB required by HARP.
  arXiv  Detail & Related papers  (2025-06-30T09:09:50Z)
• SGLP: A Similarity Guided Fast Layer Partition Pruning for Compressing Large Deep Models [19.479746878680707]
  Layer pruning is a potent approach to reduce network size and improve computational efficiency. We propose a Similarity Guided fast Layer Partition pruning for compressing large deep models. Our method outperforms the state-of-the-art methods in both accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2024-10-14T04:01:08Z)
• LayerMerge: Neural Network Depth Compression through Layer Pruning and Merging [20.774060844559838]
  Existing depth compression methods remove redundant non-linear activation functions and merge the consecutive convolution layers into a single layer. These methods suffer from a critical drawback; the kernel size of the merged layers becomes larger. We show that this problem can be addressed by jointly pruning convolution layers and activation functions. We propose LayerMerge, a novel depth compression method that selects which activation layers and convolution layers to remove.
  arXiv  Detail & Related papers  (2024-06-18T17:55:15Z)
• LaCoOT: Layer Collapse through Optimal Transport [9.715121047425262]
  We present an optimal transport-based method to reduce the depth of over-parametrized deep neural networks.<n>We show that our method achieves better performance/depth trade-off compared to existing techniques.
  arXiv  Detail & Related papers  (2024-06-13T09:03:53Z)
• Robust Stochastically-Descending Unrolled Networks [85.6993263983062]
  Deep unrolling is an emerging learning-to-optimize method that unrolls a truncated iterative algorithm in the layers of a trainable neural network.<n>We show that convergence guarantees and generalizability of the unrolled networks are still open theoretical problems.<n>We numerically assess unrolled architectures trained under the proposed constraints in two different applications.
  arXiv  Detail & Related papers  (2023-12-25T18:51:23Z)
• Poly-NL: Linear Complexity Non-local Layers with Polynomials [76.21832434001759]
  We formulate novel fast NonLocal blocks, capable of reducing complexity from quadratic to linear with no loss in performance. The proposed method, which we dub as "Poly-NL", is competitive with state-of-the-art performance across image recognition, instance segmentation, and face detection tasks.
  arXiv  Detail & Related papers  (2021-07-06T19:51:37Z)
• DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning [79.89085533866071]
  This paper introduces DeepReduce, a versatile framework for the compressed communication of sparse tensors. DeepReduce decomposes tensors in two sets, values and indices, and allows both independent and combined compression of these sets. Our experiments with large real models demonstrate that DeepReduce transmits fewer data and imposes lower computational overhead than existing methods.
  arXiv  Detail & Related papers  (2021-02-05T11:31:24Z)
• Comprehensive Online Network Pruning via Learnable Scaling Factors [3.274290296343038]
  Deep CNNs can either be pruned width-wise by removing filters based on their importance or depth-wise by removing layers and blocks. We propose a comprehensive pruning strategy that can perform both width-wise as well as depth-wise pruning.
  arXiv  Detail & Related papers  (2020-10-06T11:04:17Z)
• Perceptron Synthesis Network: Rethinking the Action Scale Variances in Videos [48.57686258913474]
  Video action recognition has been partially addressed by the CNNs stacking of fixed-size 3D kernels. We propose to learn the optimal-scale kernels from the data. An textitaction perceptron synthesizer is proposed to generate the kernels from a bag of fixed-size kernels.
  arXiv  Detail & Related papers  (2020-07-22T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.