Learned Low Precision Graph Neural Networks
- URL: http://arxiv.org/abs/2009.09232v1
- Date: Sat, 19 Sep 2020 13:51:09 GMT
- Title: Learned Low Precision Graph Neural Networks
- Authors: Yiren Zhao, Duo Wang, Daniel Bates, Robert Mullins, Mateja Jamnik,
Pietro Lio
- Abstract summary: We show how to systematically quantise Deep Graph Neural Networks (GNNs) with minimal or no loss in performance using Network Architecture Search (NAS)
The proposed novel NAS mechanism, named Low Precision Graph NAS (LPGNAS), constrains both architecture and quantisation choices to be differentiable.
On eight different datasets, solving the task of classifying unseen nodes in a graph, LPGNAS generates quantised models with significant reductions in both model and buffer sizes.
- Score: 10.269500440688306
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep Graph Neural Networks (GNNs) show promising performance on a range of
graph tasks, yet at present are costly to run and lack many of the
optimisations applied to DNNs. We show, for the first time, how to
systematically quantise GNNs with minimal or no loss in performance using
Network Architecture Search (NAS). We define the possible quantisation search
space of GNNs. The proposed novel NAS mechanism, named Low Precision Graph NAS
(LPGNAS), constrains both architecture and quantisation choices to be
differentiable. LPGNAS learns the optimal architecture coupled with the best
quantisation strategy for different components in the GNN automatically using
back-propagation in a single search round. On eight different datasets, solving
the task of classifying unseen nodes in a graph, LPGNAS generates quantised
models with significant reductions in both model and buffer sizes but with
similar accuracy to manually designed networks and other NAS results. In
particular, on the Pubmed dataset, LPGNAS shows a better size-accuracy Pareto
frontier compared to seven other manual and searched baselines, offering a 2.3
times reduction in model size but a 0.4% increase in accuracy when compared to
the best NAS competitor. Finally, from our collected quantisation statistics on
a wide range of datasets, we suggest a W4A8 (4-bit weights, 8-bit activations)
quantisation strategy might be the bottleneck for naive GNN quantisations.
Related papers
- DiffusionNAG: Predictor-guided Neural Architecture Generation with Diffusion Models [56.584561770857306]
We propose a novel conditional Neural Architecture Generation (NAG) framework based on diffusion models, dubbed DiffusionNAG.
Specifically, we consider the neural architectures as directed graphs and propose a graph diffusion model for generating them.
We validate the effectiveness of DiffusionNAG through extensive experiments in two predictor-based NAS scenarios: Transferable NAS and Bayesian Optimization (BO)-based NAS.
When integrated into a BO-based algorithm, DiffusionNAG outperforms existing BO-based NAS approaches, particularly in the large MobileNetV3 search space on the ImageNet 1K dataset.
arXiv Detail & Related papers (2023-05-26T13:58:18Z) - Do Not Train It: A Linear Neural Architecture Search of Graph Neural
Networks [15.823247346294089]
We develop a novel NAS-GNNs method, namely neural architecture coding (NAC)
Our approach leads to state-of-the-art performance, which is up to $200times$ faster and $18.8%$ more accurate than the strong baselines.
arXiv Detail & Related papers (2023-05-23T13:44:04Z) - A Comprehensive Study on Large-Scale Graph Training: Benchmarking and
Rethinking [124.21408098724551]
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs)
We present a new ensembling training manner, named EnGCN, to address the existing issues.
Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
arXiv Detail & Related papers (2022-10-14T03:43:05Z) - GHN-Q: Parameter Prediction for Unseen Quantized Convolutional
Architectures via Graph Hypernetworks [80.29667394618625]
We conduct the first-ever study exploring the use of graph hypernetworks for predicting parameters of unseen quantized CNN architectures.
We focus on a reduced CNN search space and find that GHN-Q can in fact predict quantization-robust parameters for various 8-bit quantized CNNs.
arXiv Detail & Related papers (2022-08-26T08:00:02Z) - Sub-bit Neural Networks: Learning to Compress and Accelerate Binary
Neural Networks [72.81092567651395]
Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs.
SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space.
Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
arXiv Detail & Related papers (2021-10-18T11:30:29Z) - Search to aggregate neighborhood for graph neural network [47.47628113034479]
We propose a framework, which tries to Search to Aggregate NEighborhood (SANE) to automatically design data-specific GNN architectures.
By designing a novel and expressive search space, we propose a differentiable search algorithm, which is more efficient than previous reinforcement learning based methods.
arXiv Detail & Related papers (2021-04-14T03:15:19Z) - PEng4NN: An Accurate Performance Estimation Engine for Efficient
Automated Neural Network Architecture Search [0.0]
Neural network (NN) models are increasingly used in scientific simulations, AI, and other high performance computing fields.
NAS attempts to find well-performing NN models for specialized datsets, where performance is measured by key metrics that capture the NN capabilities.
We propose a performance estimation strategy that reduces the resources for training NNs and increases NAS throughput without jeopardizing accuracy.
arXiv Detail & Related papers (2021-01-11T20:49:55Z) - Degree-Quant: Quantization-Aware Training for Graph Neural Networks [10.330195866109312]
Graph neural networks (GNNs) have demonstrated strong performance on a wide variety of tasks.
Despite their promise, there exists little research exploring methods to make them more efficient at inference time.
We propose an architecturally-agnostic method, Degree-Quant, to improve performance over existing quantization-aware training baselines.
arXiv Detail & Related papers (2020-08-11T20:53:50Z) - SGQuant: Squeezing the Last Bit on Graph Neural Networks with
Specialized Quantization [16.09107108787834]
We propose a specialized GNN quantization scheme, SGQuant, to systematically reduce the GNN memory consumption.
We show that SGQuant can effectively reduce the memory footprint from 4.25x to 31.9x compared with the original full-precision GNNs.
arXiv Detail & Related papers (2020-07-09T22:42:34Z) - Binarized Graph Neural Network [65.20589262811677]
We develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters.
Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches.
Experiments indicate that the proposed binarized graph neural network, namely BGN, is orders of magnitude more efficient in terms of both time and space.
arXiv Detail & Related papers (2020-04-19T09:43:14Z) - Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters.
Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques.
We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
arXiv Detail & Related papers (2020-02-03T04:11:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.