Pruning Convolutional Filters via Reinforcement Learning with Entropy Minimization

• URL: http://arxiv.org/abs/2312.04918v1
• Date: Fri, 8 Dec 2023 09:34:57 GMT
• Title: Pruning Convolutional Filters via Reinforcement Learning with Entropy Minimization
• Authors: Bogdan Musat, Razvan Andonie
• Abstract summary: We introduce a novel information-theoretic reward function which minimizes the spatial entropy of convolutional activations. Our method shows that there is another possibility to preserve accuracy without the need to directly optimize it in the agent's reward function.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Structural pruning has become an integral part of neural network optimization, used to achieve architectural configurations which can be deployed and run more efficiently on embedded devices. Previous results showed that pruning is possible with minimum performance loss by utilizing a reinforcement learning agent which makes decisions about the sparsity level of each neural layer by maximizing as a reward the accuracy of the network. We introduce a novel information-theoretic reward function which minimizes the spatial entropy of convolutional activations. This minimization ultimately acts as a proxy for maintaining accuracy, although these two criteria are not related in any way. Our method shows that there is another possibility to preserve accuracy without the need to directly optimize it in the agent's reward function. In our experiments, we were able to reduce the total number of FLOPS of multiple popular neural network architectures by 5-10x, incurring minimal or no performance drop and being on par with the solution found by maximizing the accuracy.

Related papers

• Just How Flexible are Neural Networks in Practice? [89.80474583606242]
  It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters. In practice, however, we only find solutions via our training procedure, including the gradient and regularizers, limiting flexibility.
  arXiv  Detail & Related papers  (2024-06-17T12:24:45Z)
• Growing Tiny Networks: Spotting Expressivity Bottlenecks and Fixing Them Optimally [2.645067871482715]
  In machine learning tasks, one searches for an optimal function within a certain functional space. This way forces the evolution of the function during training to lie within the realm of what is expressible with the chosen architecture. We show that the information about desirable architectural changes, due to expressivity bottlenecks can be extracted from %the backpropagation.
  arXiv  Detail & Related papers  (2024-05-30T08:23:56Z)
• Improving Generalization of Deep Neural Networks by Optimum Shifting [33.092571599896814]
  We propose a novel method called emphoptimum shifting, which changes the parameters of a neural network from a sharp minimum to a flatter one. Our method is based on the observation that when the input and output of a neural network are fixed, the matrix multiplications within the network can be treated as systems of under-determined linear equations.
  arXiv  Detail & Related papers  (2024-05-23T02:31:55Z)
• Explicit Foundation Model Optimization with Self-Attentive Feed-Forward Neural Units [4.807347156077897]
  Iterative approximation methods using backpropagation enable the optimization of neural networks, but they remain computationally expensive when used at scale. This paper presents an efficient alternative for optimizing neural networks that reduces the costs of scaling neural networks and provides high-efficiency optimizations for low-resource applications.
  arXiv  Detail & Related papers  (2023-11-13T17:55:07Z)
• On Model Compression for Neural Networks: Framework, Algorithm, and Convergence Guarantee [21.818773423324235]
  This paper focuses on two model compression techniques: low-rank approximation and weight approximation. In this paper, a holistic framework is proposed for model compression from a novel perspective of non optimization.
  arXiv  Detail & Related papers  (2023-03-13T02:14:42Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Fast Exploration of the Impact of Precision Reduction on Spiking Neural Networks [63.614519238823206]
  Spiking Neural Networks (SNNs) are a practical choice when the target hardware reaches the edge of computing. We employ an Interval Arithmetic (IA) model to develop an exploration methodology that takes advantage of the capability of such a model to propagate the approximation error.
  arXiv  Detail & Related papers  (2022-11-22T15:08:05Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning [83.59005356327103]
  A common limitation of most existing pruning techniques is that they require pre-training of the network at least once before pruning. We propose STAMP, which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. We validate STAMP against recent advanced pruning methods on benchmark datasets.
  arXiv  Detail & Related papers  (2020-06-22T10:57:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.