Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networks

• URL: http://arxiv.org/abs/2312.15229v1
• Date: Sat, 23 Dec 2023 11:37:18 GMT
• Title: Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networks
• Authors: Toluwani Aremu
• Abstract summary: We focus on PolyKervNets, a technique known for offering improved dynamic approximations in smaller networks. Our primary objective is to empirically explore optimization-based training recipes to enhance the performance of PolyKervNets in larger networks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Private computation of nonlinear functions, such as Rectified Linear Units (ReLUs) and max-pooling operations, in deep neural networks (DNNs) poses significant challenges in terms of storage, bandwidth, and time consumption. To address these challenges, there has been a growing interest in utilizing privacy-preserving techniques that leverage polynomial activation functions and kernelized convolutions as alternatives to traditional ReLUs. However, these alternative approaches often suffer from a trade-off between achieving faster private inference (PI) and sacrificing model accuracy. In particular, when applied to much deeper networks, these methods encounter training instabilities, leading to issues like exploding gradients (resulting in NaNs) or suboptimal approximations. In this study, we focus on PolyKervNets, a technique known for offering improved dynamic approximations in smaller networks but still facing instabilities in larger and more complex networks. Our primary objective is to empirically explore optimization-based training recipes to enhance the performance of PolyKervNets in larger networks. By doing so, we aim to potentially eliminate the need for traditional nonlinear activation functions, thereby advancing the state-of-the-art in privacy-preserving deep neural network architectures. Code can be found on GitHub at: \url{https://github.com/tolusophy/PolyKervNets/}

Related papers

• RandONet: Shallow-Networks with Random Projections for learning linear and nonlinear operators [0.0]
  We present Random Projection-based Operator Networks (RandONets) RandONets are shallow networks with random projections that learn linear and nonlinear operators. We show, that for this particular task, RandONets outperform, both in terms of numerical approximation accuracy and computational cost, the vanilla" DeepOnets.
  arXiv  Detail & Related papers  (2024-06-08T13:20:48Z)
• Fixing the NTK: From Neural Network Linearizations to Exact Convex Programs [63.768739279562105]
  We show that for a particular choice of mask weights that do not depend on the learning targets, this kernel is equivalent to the NTK of the gated ReLU network on the training data. A consequence of this lack of dependence on the targets is that the NTK cannot perform better than the optimal MKL kernel on the training set.
  arXiv  Detail & Related papers  (2023-09-26T17:42:52Z)
• Regularization of polynomial networks for image recognition [78.4786845859205]
  Polynomial Networks (PNs) have emerged as an alternative method with a promising performance and improved interpretability. We introduce a class of PNs, which are able to reach the performance of ResNet across a range of six benchmarks.
  arXiv  Detail & Related papers  (2023-03-24T10:05:22Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Learning k-Level Structured Sparse Neural Networks Using Group Envelope Regularization [4.0554893636822]
  We introduce a novel approach to deploy large-scale Deep Neural Networks on constrained resources. The method speeds up inference time and aims to reduce memory demand and power consumption.
  arXiv  Detail & Related papers  (2022-12-25T15:40:05Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Edge Rewiring Goes Neural: Boosting Network Resilience via Policy Gradient [62.660451283548724]
  ResiNet is a reinforcement learning framework to discover resilient network topologies against various disasters and attacks. We show that ResiNet achieves a near-optimal resilience gain on multiple graphs while balancing the utility, with a large margin compared to existing approaches.
  arXiv  Detail & Related papers  (2021-10-18T06:14:28Z)
• Wide Network Learning with Differential Privacy [7.453881927237143]
  Current generation of neural networks suffers significant loss accuracy under most practically relevant privacy training regimes. We develop a general approach towards training these models that takes advantage of the sparsity of the gradients of private Empirical Minimization (ERM) Following the same number of parameters, we propose a novel algorithm for privately training neural networks.
  arXiv  Detail & Related papers  (2021-03-01T20:31:50Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• Streaming Probabilistic Deep Tensor Factorization [27.58928876734886]
  We propose SPIDER, a Streaming ProbabilistIc Deep tEnsoR factorization method. We develop an efficient streaming posterior inference algorithm in the assumed-density-filtering and expectation propagation framework. We show the advantages of our approach in four real-world applications.
  arXiv  Detail & Related papers  (2020-07-14T21:25:39Z)
• Lossless Compression of Deep Neural Networks [17.753357839478575]
  Deep neural networks have been successful in many predictive modeling tasks, such as image and language recognition. It is challenging to deploy these networks under limited computational resources, such as in mobile devices. We introduce an algorithm that removes units and layers of a neural network while not changing the output that is produced.
  arXiv  Detail & Related papers  (2020-01-01T15:04:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.