Householder Activations for Provable Robustness against Adversarial Attacks

• URL: http://arxiv.org/abs/2108.04062v1
• Date: Thu, 5 Aug 2021 12:02:16 GMT
• Title: Householder Activations for Provable Robustness against Adversarial Attacks
• Authors: Sahil Singla, Surbhi Singla, Soheil Feizi
• Abstract summary: Training convolutional neural networks (CNNs) with a strict Lipschitz constraint under the l_2 norm is useful for provable adversarial robustness, interpretable gradients and stable training. We introduce a class of nonlinear GNP activations with learnable Householder transformations called Householder activations. Our experiments on CIFAR-10 and CIFAR-100 show that our regularized networks with $mathrmHH$ activations lead to significant improvements in both the standard and provable robust accuracy.
• Score: 37.289891549908596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training convolutional neural networks (CNNs) with a strict Lipschitz constraint under the l_{2} norm is useful for provable adversarial robustness, interpretable gradients and stable training. While 1-Lipschitz CNNs can be designed by enforcing a 1-Lipschitz constraint on each layer, training such networks requires each layer to have an orthogonal Jacobian matrix (for all inputs) to prevent gradients from vanishing during backpropagation. A layer with this property is said to be Gradient Norm Preserving (GNP). To construct expressive GNP activation functions, we first prove that the Jacobian of any GNP piecewise linear function is only allowed to change via Householder transformations for the function to be continuous. Building on this result, we introduce a class of nonlinear GNP activations with learnable Householder transformations called Householder activations. A householder activation parameterized by the vector $\mathbf{v}$ outputs $(\mathbf{I} - 2\mathbf{v}\mathbf{v}^{T})\mathbf{z}$ for its input $\mathbf{z}$ if $\mathbf{v}^{T}\mathbf{z} \leq 0$; otherwise it outputs $\mathbf{z}$. Existing GNP activations such as $\mathrm{MaxMin}$ can be viewed as special cases of $\mathrm{HH}$ activations for certain settings of these transformations. Thus, networks with $\mathrm{HH}$ activations have higher expressive power than those with $\mathrm{MaxMin}$ activations. Although networks with $\mathrm{HH}$ activations have nontrivial provable robustness against adversarial attacks, we further boost their robustness by (i) introducing a certificate regularization and (ii) relaxing orthogonalization of the last layer of the network. Our experiments on CIFAR-10 and CIFAR-100 show that our regularized networks with $\mathrm{HH}$ activations lead to significant improvements in both the standard and provable robust accuracy over the prior works (gain of 3.65\% and 4.46\% on CIFAR-100 respectively).

Related papers

• Deep Neural Network Initialization with Sparsity Inducing Activations [5.437298646956505]
  We use the large width Gaussian process limit to analyze the behaviour of nonlinear activations that induce sparsity in the hidden outputs. A previously unreported form of training instability is proven for arguably two of the most natural candidates for hidden layer sparsification. We show that this instability is overcome by clipping the nonlinear activation magnitude, at a level prescribed by the shape of the associated Gaussian process variance map.
  arXiv  Detail & Related papers  (2024-02-25T20:11:40Z)
• Improved techniques for deterministic l2 robustness [63.34032156196848]
  Training convolutional neural networks (CNNs) with a strict 1-Lipschitz constraint under the $l_2$ norm is useful for adversarial robustness, interpretable gradients and stable training. We introduce a procedure to certify robustness of 1-Lipschitz CNNs by replacing the last linear layer with a 1-hidden layer. We significantly advance the state-of-the-art for standard and provable robust accuracies on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2022-11-15T19:10:12Z)
• Learning a Single Neuron with Adversarial Label Noise via Gradient Descent [50.659479930171585]
  We study a function of the form $mathbfxmapstosigma(mathbfwcdotmathbfx)$ for monotone activations. The goal of the learner is to output a hypothesis vector $mathbfw$ that $F(mathbbw)=C, epsilon$ with high probability.
  arXiv  Detail & Related papers  (2022-06-17T17:55:43Z)
• Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations [27.244958998196623]
  We present time and sample efficient algorithms for learning an unknown depth-2 feedforward neural network with general ReLU activations. In particular, we consider learning an unknown network of the form $f(x) = amathsfTsigma(WmathsfTx+b)$, where $x$ is drawn from the Gaussian distribution, and $sigma(t) := max(t,0)$ is the ReLU activation.
  arXiv  Detail & Related papers  (2021-07-21T17:06:03Z)
• Skew Orthogonal Convolutions [44.053067014796596]
  Training convolutional neural networks with a Lipschitz constraint under the $l_2$ norm is useful for provable adversarial robustness, interpretable gradients, stable training, etc. Methodabv allows us to train provably Lipschitz, large convolutional neural networks significantly faster than prior works.
  arXiv  Detail & Related papers  (2021-05-24T17:11:44Z)
• Beyond Lazy Training for Over-parameterized Tensor Decomposition [69.4699995828506]
  We show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data. Our results show that gradient descent on over-parametrized objective could go beyond the lazy training regime and utilize certain low-rank structure in the data.
  arXiv  Detail & Related papers  (2020-10-22T00:32:12Z)
• Agnostic Learning of a Single Neuron with Gradient Descent [92.7662890047311]
  We consider the problem of learning the best-fitting single neuron as measured by the expected square loss. For the ReLU activation, our population risk guarantee is $O(mathsfOPT1/2)+epsilon$. For the ReLU activation, our population risk guarantee is $O(mathsfOPT1/2)+epsilon$.
  arXiv  Detail & Related papers  (2020-05-29T07:20:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.