DepGraph: Towards Any Structural Pruning

• URL: http://arxiv.org/abs/2301.12900v2
• Date: Thu, 23 Mar 2023 12:55:02 GMT
• Title: DepGraph: Towards Any Structural Pruning
• Authors: Gongfan Fang, Xinyin Ma, Mingli Song, Michael Bi Mi, Xinchao Wang
• Abstract summary: We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
• Score: 68.40343338847664
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Structural pruning enables model acceleration by removing structurally-grouped parameters from neural networks. However, the parameter-grouping patterns vary widely across different models, making architecture-specific pruners, which rely on manually-designed grouping schemes, non-generalizable to new architectures. In this work, we study a highly-challenging yet barely-explored task, any structural pruning, to tackle general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. The most prominent obstacle towards this goal lies in the structural coupling, which not only forces different layers to be pruned simultaneously, but also expects all removed parameters to be consistently unimportant, thereby avoiding structural issues and significant performance degradation after pruning. To address this problem, we propose a general and {fully automatic} method, \emph{Dependency Graph} (DepGraph), to explicitly model the dependency between layers and comprehensively group coupled parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a simple norm-based criterion, the proposed method consistently yields gratifying performances.

Related papers

• Isomorphic Pruning for Vision Models [56.286064975443026]
  Structured pruning reduces the computational overhead of deep neural networks by removing redundant sub-structures. We present Isomorphic Pruning, a simple approach that demonstrates effectiveness across a range of network architectures.
  arXiv  Detail & Related papers  (2024-07-05T16:14:53Z)
• Structurally Prune Anything: Any Architecture, Any Framework, Any Time [84.6210631783801]
  We introduce Structurally Prune Anything (SPA), a versatile structured pruning framework for neural networks. SPA supports pruning at any time, either before training, after training with fine-tuning, or after training without fine-tuning. In extensive experiments, SPA shows competitive to state-of-the-art pruning performance across various architectures.
  arXiv  Detail & Related papers  (2024-03-03T13:49:49Z)
• Distance Weighted Trans Network for Image Completion [52.318730994423106]
  We propose a new architecture that relies on Distance-based Weighted Transformer (DWT) to better understand the relationships between an image's components. CNNs are used to augment the local texture information of coarse priors. DWT blocks are used to recover certain coarse textures and coherent visual structures.
  arXiv  Detail & Related papers  (2023-10-11T12:46:11Z)
• ASGNN: Graph Neural Networks with Adaptive Structure [41.83813812538167]
  We propose a novel interpretable message passing scheme with adaptive structure (ASMP) to defend against adversarial attacks on graph structure. ASMP is adaptive in the sense that the message passing process in different layers is able to be carried out over dynamically adjusted graphs.
  arXiv  Detail & Related papers  (2022-10-03T15:10:40Z)
• Deep Neural Networks pruning via the Structured Perspective Regularization [5.061851539114448]
  In Machine Learning, Artificial Neural Networks (ANNs) are a very powerful tool, broadly used in many applications. One of the most popular compression approaches is emphpruning, whereby entire elements of the ANN (links, nodes, channels, ldots) and the corresponding weights are deleted. Since the nature of the problem is inherently (what elements to prune and what not), we propose a new pruning method based on Operational Research tools.
  arXiv  Detail & Related papers  (2022-06-28T14:58:51Z)
• Inter-layer Transition in Neural Architecture Search [89.00449751022771]
  The dependency between the architecture weights of connected edges is explicitly modeled in this paper. Experiments on five benchmarks confirm the value of modeling inter-layer dependency and demonstrate the proposed method outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-11-30T03:33:52Z)
• Adversarially Robust Neural Architectures [43.74185132684662]
  This paper aims to improve the adversarial robustness of the network from the architecture perspective with NAS framework. We explore the relationship among adversarial robustness, Lipschitz constant, and architecture parameters. Our algorithm empirically achieves the best performance among all the models under various attacks on different datasets.
  arXiv  Detail & Related papers  (2020-09-02T08:52:15Z)
• Structured Convolutions for Efficient Neural Network Design [65.36569572213027]
  We tackle model efficiency by exploiting redundancy in the textitimplicit structure of the building blocks of convolutional neural networks. We show how this decomposition can be applied to 2D and 3D kernels as well as the fully-connected layers.
  arXiv  Detail & Related papers  (2020-08-06T04:38:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.