Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning

• URL: http://arxiv.org/abs/2107.12342v1
• Date: Mon, 26 Jul 2021 17:33:56 GMT
• Title: Sisyphus: A Cautionary Tale of Using Low-Degree Polynomial Activations in Privacy-Preserving Deep Learning
• Authors: Karthik Garimella, Nandan Kumar Jha and Brandon Reagen
• Abstract summary: Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on encrypted inputs. We ask: Is it feasible to substitute all ReLUs with low-degree bandwidth activation functions for building deep, privacy-friendly neural networks? We analyze challenges of substituting ReLUs with PIs, starting with simple drop-and-replace solutions to novel, more involved replace-and-retrain strategies.
• Score: 2.5677092608889773
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Privacy concerns in client-server machine learning have given rise to private inference (PI), where neural inference occurs directly on encrypted inputs. PI protects clients' personal data and the server's intellectual property. A common practice in PI is to use garbled circuits to compute nonlinear functions privately, namely ReLUs. However, garbled circuits suffer from high storage, bandwidth, and latency costs. To mitigate these issues, PI-friendly polynomial activation functions have been employed to replace ReLU. In this work, we ask: Is it feasible to substitute all ReLUs with low-degree polynomial activation functions for building deep, privacy-friendly neural networks? We explore this question by analyzing the challenges of substituting ReLUs with polynomials, starting with simple drop-and-replace solutions to novel, more involved replace-and-retrain strategies. We examine the limitations of each method and provide commentary on the use of polynomial activation functions for PI. We find all evaluated solutions suffer from the escaping activation problem: forward activation values inevitably begin to expand at an exponential rate away from stable regions of the polynomials, which leads to exploding values (NaNs) or poor approximations.

Related papers

• Regularized PolyKervNets: Optimizing Expressiveness and Efficiency for Private Inference in Deep Neural Networks [0.0]
  We focus on PolyKervNets, a technique known for offering improved dynamic approximations in smaller networks. Our primary objective is to empirically explore optimization-based training recipes to enhance the performance of PolyKervNets in larger networks.
  arXiv  Detail & Related papers  (2023-12-23T11:37:18Z)
• Physics-informed PointNet: On how many irregular geometries can it solve an inverse problem simultaneously? Application to linear elasticity [58.44709568277582]
  Physics-informed PointNet (PIPN) is designed to fill this gap between PINNs and fully supervised learning models. We show that PIPN predicts the solution of desired partial differential equations over a few hundred domains simultaneously. Specifically, we show that PIPN predicts the solution of a plane stress problem over more than 500 domains with different geometries, simultaneously.
  arXiv  Detail & Related papers  (2023-03-22T06:49:34Z)
• Selective Network Linearization for Efficient Private Inference [49.937470642033155]
  We propose a gradient-based algorithm that selectively linearizes ReLUs while maintaining prediction accuracy. The results demonstrate up to $4.25%$ more accuracy (iso-ReLU count at 50K) or $2.2times$ less latency (iso-accuracy at 70%) than the current state of the art.
  arXiv  Detail & Related papers  (2022-02-04T19:00:24Z)
• Circa: Stochastic ReLUs for Private Deep Learning [6.538025863698682]
  We re-think the ReLU computation and propose optimizations for PI tailored to neural networks. Specifically, we reformulate ReLU as an approximate sign test and introduce a novel truncation method for the sign test. We demonstrate improvements of up to 4.7x storage and 3x runtime over baseline implementations.
  arXiv  Detail & Related papers  (2021-06-15T22:52:45Z)
• S++: A Fast and Deployable Secure-Computation Framework for Privacy-Preserving Neural Network Training [0.4893345190925178]
  We introduce S++, a simple, robust, and deployable framework for training a neural network (NN) using private data from multiple sources. For the first time, we provide fast and verifiable protocols for all common activation functions and optimize them for running in a secret-shared manner.
  arXiv  Detail & Related papers  (2021-01-28T15:48:54Z)
• On Polynomial Approximations for Privacy-Preserving and Verifiable ReLU Networks [6.130998208629276]
  We propose a degree-2 activation function with a first order term and empirically show that it can lead to much better models. Our proposed function improves the test accuracy by up to 10.4% compared to the square function.
  arXiv  Detail & Related papers  (2020-11-11T03:32:22Z)
• Activation Relaxation: A Local Dynamical Approximation to Backpropagation in the Brain [62.997667081978825]
  Activation Relaxation (AR) is motivated by constructing the backpropagation gradient as the equilibrium point of a dynamical system. Our algorithm converges rapidly and robustly to the correct backpropagation gradients, requires only a single type of computational unit, and can operate on arbitrary computation graphs.
  arXiv  Detail & Related papers  (2020-09-11T11:56:34Z)
• HiPPO: Recurrent Memory with Optimal Polynomial Projections [93.3537706398653]
  We introduce a general framework (HiPPO) for the online compression of continuous signals and discrete time series by projection onto bases. Given a measure that specifies the importance of each time step in the past, HiPPO produces an optimal solution to a natural online function approximation problem. This formal framework yields a new memory update mechanism (HiPPO-LegS) that scales through time to remember all history, avoiding priors on the timescale.
  arXiv  Detail & Related papers  (2020-08-17T23:39:33Z)
• Computational Barriers to Estimation from Low-Degree Polynomials [81.67886161671379]
  We study the power of low-degrees for the task of detecting the presence of hidden structures. For a large class of "signal plus noise" problems, we give a user-friendly lower bound for the best possible mean squared error achievable by any degree. As applications, we give a tight characterization of the low-degree minimum mean squared error for the planted submatrix and planted dense subgraph problems.
  arXiv  Detail & Related papers  (2020-08-05T17:52:10Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)
• Deep Neural Networks with Trainable Activations and Controlled Lipschitz Constant [26.22495169129119]
  We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the Lipschitz constant. We numerically compare our scheme with standard ReLU network and its variations, PReLU and LeakyReLU.
  arXiv  Detail & Related papers  (2020-01-17T12:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.