Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning

• URL: http://arxiv.org/abs/2210.00053v1
• Date: Fri, 30 Sep 2022 19:33:53 GMT
• Title: Kernel Normalized Convolutional Networks for Privacy-Preserving Machine Learning
• Authors: Reza Nasirigerdeh, Javad Torkzadehmahani, Daniel Rueckert, Georgios Kaissis
• Abstract summary: We compare layer normalization (LayerNorm), group normalization (GroupNorm), and the recently proposed kernel normalization ( KernelNorm) in FL and DP settings. LayerNorm and GroupNorm provide no performance gain compared to the baseline (i.e. no normalization) for shallow models, but they considerably enhance performance of deeper models. KernelNorm, on the other hand, significantly outperforms its competitors in terms of accuracy and convergence rate (or communication efficiency) for both shallow and deeper models.
• Score: 7.384030323608299
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Normalization is an important but understudied challenge in privacy-related application domains such as federated learning (FL) and differential privacy (DP). While the unsuitability of batch normalization for FL and DP has already been shown, the impact of the other normalization methods on the performance of federated or differentially private models is not well-known. To address this, we draw a performance comparison among layer normalization (LayerNorm), group normalization (GroupNorm), and the recently proposed kernel normalization (KernelNorm) in FL and DP settings. Our results indicate LayerNorm and GroupNorm provide no performance gain compared to the baseline (i.e. no normalization) for shallow models, but they considerably enhance performance of deeper models. KernelNorm, on the other hand, significantly outperforms its competitors in terms of accuracy and convergence rate (or communication efficiency) for both shallow and deeper models. Given these key observations, we propose a kernel normalized ResNet architecture called KNResNet-13 for differentially private learning environments. Using the proposed architecture, we provide new state-of-the-art accuracy values on the CIFAR-10 and Imagenette datasets.

Related papers

• Differentially Private Learning with Per-Sample Adaptive Clipping [8.401653565794353]
  We propose a Differentially Private Per-Sample Adaptive Clipping (DP-PSAC) algorithm based on a non-monotonic adaptive weight function. We show that DP-PSAC outperforms or matches the state-of-the-art methods on multiple main-stream vision and language tasks.
  arXiv  Detail & Related papers  (2022-12-01T07:26:49Z)
• Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
  We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter. We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures'' Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
  arXiv  Detail & Related papers  (2022-10-23T18:37:22Z)
• Rethinking Normalization Methods in Federated Learning [92.25845185724424]
  Federated learning (FL) is a popular distributed learning framework that can reduce privacy risks by not explicitly sharing private data. We show that external covariate shifts will lead to the obliteration of some devices' contributions to the global model.
  arXiv  Detail & Related papers  (2022-10-07T01:32:24Z)
• Kernel Normalized Convolutional Networks [15.997774467236352]
  BatchNorm, however, performs poorly with small batch sizes, and is inapplicable to differential privacy. We propose KernelNorm and kernel normalized convolutional layers, and incorporate them into kernel normalized convolutional networks (KNConvNets) KNConvNets achieve higher or competitive performance compared to BatchNorm counterparts in image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2022-05-20T11:18:05Z)
• SmoothNets: Optimizing CNN architecture design for differentially private deep learning [69.10072367807095]
  DPSGD requires clipping and noising of per-sample gradients. This introduces a reduction in model utility compared to non-private training. We distilled a new model architecture termed SmoothNet, which is characterised by increased robustness to the challenges of DP-SGD training.
  arXiv  Detail & Related papers  (2022-05-09T07:51:54Z)
• Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy [67.4471689755097]
  This paper empirically demonstrates that the clipped FedAvg can perform surprisingly well even with substantial data heterogeneity. We provide the convergence analysis of a differential private (DP) FedAvg algorithm and highlight the relationship between clipping bias and the distribution of the clients' updates.
  arXiv  Detail & Related papers  (2021-06-25T14:47:19Z)
• Correct Normalization Matters: Understanding the Effect of Normalization On Deep Neural Network Models For Click-Through Rate Prediction [3.201333208812837]
  We propose a new and effective normalization approaches based on LayerNorm named variance only LayerNorm(VO-LN) in this work. We find that the variance of normalization plays the main role and give an explanation in this work.
  arXiv  Detail & Related papers  (2020-06-23T04:35:22Z)
• Understanding and Resolving Performance Degradation in Graph Convolutional Networks [105.14867349802898]
  Graph Convolutional Network (GCN) stacks several layers and in each layer performs a PROPagation operation (PROP) and a TRANsformation operation (TRAN) for learning node representations over graph-structured data. GCNs tend to suffer performance drop when the model gets deep. We study performance degradation of GCNs by experimentally examining how stacking only TRANs or PROPs works.
  arXiv  Detail & Related papers  (2020-06-12T12:12:12Z)
• Iterative Network for Image Super-Resolution [69.07361550998318]
  Single image super-resolution (SISR) has been greatly revitalized by the recent development of convolutional neural networks (CNN) This paper provides a new insight on conventional SISR algorithm, and proposes a substantially different approach relying on the iterative optimization. A novel iterative super-resolution network (ISRN) is proposed on top of the iterative optimization.
  arXiv  Detail & Related papers  (2020-05-20T11:11:47Z)
• Rethinking Depthwise Separable Convolutions: How Intra-Kernel Correlations Lead to Improved MobileNets [6.09170287691728]
  We introduce blueprint separable convolutions (BSConv) as highly efficient building blocks for CNNs. They are motivated by quantitative analyses of kernel properties from trained models. Our approach provides a thorough theoretical derivation, interpretation, and justification for the application of depthwise separable convolutions.
  arXiv  Detail & Related papers  (2020-03-30T15:23:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.