Budget-Aware Graph Convolutional Network Design using Probabilistic Magnitude Pruning

• URL: http://arxiv.org/abs/2305.19343v1
• Date: Tue, 30 May 2023 18:12:13 GMT
• Title: Budget-Aware Graph Convolutional Network Design using Probabilistic Magnitude Pruning
• Authors: Hichem Sahbi
• Abstract summary: We devise a novel lightweight Graph convolutional networks (GCNs) design dubbed as Probabilistic Magnitude Pruning (PMP) Our method is variational and proceeds by aligning the weight distribution of the learned networks with a priori distribution. Experiments conducted on the challenging task of skeleton-based recognition show a substantial gain of our lightweight GCNs.
• Score: 12.18340575383456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph convolutional networks (GCNs) are nowadays becoming mainstream in solving many image processing tasks including skeleton-based recognition. Their general recipe consists in learning convolutional and attention layers that maximize classification performances. With multi-head attention, GCNs are highly accurate but oversized, and their deployment on edge devices requires their pruning. Among existing methods, magnitude pruning (MP) is relatively effective but its design is clearly suboptimal as network topology selection and weight retraining are achieved independently. In this paper, we devise a novel lightweight GCN design dubbed as Probabilistic Magnitude Pruning (PMP) that jointly trains network topology and weights. Our method is variational and proceeds by aligning the weight distribution of the learned networks with an a priori distribution. This allows implementing any fixed pruning rate, and also enhancing the generalization performances of the designed lightweight GCNs. Extensive experiments conducted on the challenging task of skeleton-based recognition show a substantial gain of our lightweight GCNs particularly at very high pruning regimes.

Related papers

• One-Shot Multi-Rate Pruning of Graph Convolutional Networks [5.656581242851759]
  We devise a novel lightweight Graph Convolutional Network (GCN) design dubbed as Multi-Rate Magnitude Pruning (MRMP) Our method is variational and proceeds by aligning the weight distribution of the learned networks with an a priori distribution. In the other hand, MRMP achieves a joint training of multiple GCNs, on top of shared weights, in order to extrapolate accurate networks at any targeted pruning rate without retraining their weights.
  arXiv  Detail & Related papers  (2023-12-29T14:20:00Z)
• T-GAE: Transferable Graph Autoencoder for Network Alignment [79.89704126746204]
  T-GAE is a graph autoencoder framework that leverages transferability and stability of GNNs to achieve efficient network alignment without retraining. Our experiments demonstrate that T-GAE outperforms the state-of-the-art optimization method and the best GNN approach by up to 38.7% and 50.8%, respectively.
  arXiv  Detail & Related papers  (2023-10-05T02:58:29Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Training Lightweight Graph Convolutional Networks with Phase-field Models [12.18340575383456]
  We design lightweight graph convolutional networks (GCNs) using a particular class of regularizers, dubbed as phase-field models (PFMs) PFMs exhibit a bi-phase behavior using a particular ultra-local term that allows training both the topology and the weight parameters of GCNs as a part of a single "end-to-end" optimization problem.
  arXiv  Detail & Related papers  (2022-12-19T12:49:03Z)
• Lightweight Graph Convolutional Networks with Topologically Consistent Magnitude Pruning [12.18340575383456]
  Graph convolution networks (GCNs) are currently mainstream in learning with irregular data. In this paper, we devise a novel method for lightweight GCN design. Our proposed approach parses and selectsworks with the highest magnitudes while guaranteeing their topological consistency.
  arXiv  Detail & Related papers  (2022-03-25T12:34:11Z)
• Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach [4.284071491453377]
  We propose a novel unstructured pruning pipeline, Attention-based Simultaneous sparse structure and Weight Learning (ASWL) ASWL proposes an efficient algorithm to calculate the pruning ratio through layer-wise attention for each layer, and both weights for the dense network and the sparse network are tracked so that the pruned structure is simultaneously learned from randomly weights. Our experiments on MNIST, Cifar10, and ImageNet show that ASWL achieves superior pruning results in terms of accuracy, pruning ratio and operating efficiency.
  arXiv  Detail & Related papers  (2021-11-01T02:27:44Z)
• Haar Wavelet Feature Compression for Quantized Graph Convolutional Networks [7.734726150561088]
  Graph Convolutional Networks (GCNs) are widely used in a variety of applications, and can be seen as an unstructured version of standard Convolutional Neural Networks (CNNs) As in CNNs, the computational cost of GCNs for large input graphs can be high and inhibit the use of these networks, especially in environments with low computational resources. We propose to utilize Haar wavelet compression and light quantization to reduce the computations and the bandwidth involved with the network.
  arXiv  Detail & Related papers  (2021-10-10T15:25:37Z)
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition [126.51241919472356]
  We design a simple and highly modularized graph convolutional network architecture for skeleton-based action recognition. Our network is constructed by repeating a building block that aggregates multi-granularity information from both the spatial and temporal paths.
  arXiv  Detail & Related papers  (2020-11-26T14:43:04Z)
• DeeperGCN: All You Need to Train Deeper GCNs [66.64739331859226]
  Graph Convolutional Networks (GCNs) have been drawing significant attention with the power of representation learning on graphs. Unlike Convolutional Neural Networks (CNNs), which are able to take advantage of stacking very deep layers, GCNs suffer from vanishing gradient, over-smoothing and over-fitting issues when going deeper. This paper proposes DeeperGCN that is capable of successfully and reliably training very deep GCNs.
  arXiv  Detail & Related papers  (2020-06-13T23:00:22Z)
• ResNeSt: Split-Attention Networks [86.25490825631763]
  We present a modularized architecture, which applies the channel-wise attention on different network branches to leverage their success in capturing cross-feature interactions and learning diverse representations. Our model, named ResNeSt, outperforms EfficientNet in accuracy and latency trade-off on image classification.
  arXiv  Detail & Related papers  (2020-04-19T20:40:31Z)
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment [84.57874289554839]
  Training deep neural networks on large-scale datasets requires significant hardware resources. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
  arXiv  Detail & Related papers  (2020-02-10T16:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.