Adversarial Robustness via Fisher-Rao Regularization
- URL: http://arxiv.org/abs/2106.06685v1
- Date: Sat, 12 Jun 2021 04:12:58 GMT
- Title: Adversarial Robustness via Fisher-Rao Regularization
- Authors: Marine Picot, Francisco Messina, Malik Boudiaf, Fabrice Labeau, Ismail
Ben Ayed, and Pablo Piantanida
- Abstract summary: Adrial robustness has become a topic of growing interest in machine learning.
Fire is a new Fisher-Rao regularization for the categorical cross-entropy loss.
- Score: 33.134075068748984
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Adversarial robustness has become a topic of growing interest in machine
learning since it was observed that neural networks tend to be brittle. We
propose an information-geometric formulation of adversarial defense and
introduce FIRE, a new Fisher-Rao regularization for the categorical
cross-entropy loss, which is based on the geodesic distance between natural and
perturbed input features. Based on the information-geometric properties of the
class of softmax distributions, we derive an explicit characterization of the
Fisher-Rao Distance (FRD) for the binary and multiclass cases, and draw some
interesting properties as well as connections with standard regularization
metrics. Furthermore, for a simple linear and Gaussian model, we show that all
Pareto-optimal points in the accuracy-robustness region can be reached by FIRE
while other state-of-the-art methods fail. Empirically, we evaluate the
performance of various classifiers trained with the proposed loss on standard
datasets, showing up to 2\% of improvements in terms of robustness while
reducing the training time by 20\% over the best-performing methods.
Related papers
- Implicit regularization via soft ascent-descent [7.335712499936906]
We show how to achieve better off-sample generalization with minimal trial-and-error.
We propose a softened, pointwise mechanism called SoftAD that downweights points on the borderline, limits the effects of outliers, and retains the ascent-descent effect.
Our empirical tests range from simple binary classification on the plane to image classification using neural networks with millions of parameters.
arXiv Detail & Related papers (2023-10-16T02:02:56Z) - Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
We propose a novel doubly-robust instance reweighted adversarial framework.
Our importance weights are obtained by optimizing the KL-divergence regularized loss function.
Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
arXiv Detail & Related papers (2023-08-01T06:16:18Z) - Variational Classification [51.2541371924591]
We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders.
Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency.
We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
arXiv Detail & Related papers (2023-05-17T17:47:19Z) - TWINS: A Fine-Tuning Framework for Improved Transferability of
Adversarial Robustness and Generalization [89.54947228958494]
This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks.
We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework.
TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
arXiv Detail & Related papers (2023-03-20T14:12:55Z) - Uncertainty Estimation by Fisher Information-based Evidential Deep
Learning [61.94125052118442]
Uncertainty estimation is a key factor that makes deep learning reliable in practical applications.
We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL)
In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
arXiv Detail & Related papers (2023-03-03T16:12:59Z) - Adaptive Dimension Reduction and Variational Inference for Transductive
Few-Shot Classification [2.922007656878633]
We propose a new clustering method based on Variational Bayesian inference, further improved by Adaptive Dimension Reduction.
Our proposed method significantly improves accuracy in the realistic unbalanced transductive setting on various Few-Shot benchmarks.
arXiv Detail & Related papers (2022-09-18T10:29:02Z) - Efficient and Robust Classification for Sparse Attacks [34.48667992227529]
We consider perturbations bounded by the $ell$--norm, which have been shown as effective attacks in the domains of image-recognition, natural language processing, and malware-detection.
We propose a novel defense method that consists of "truncation" and "adrial training"
Motivated by the insights we obtain, we extend these components to neural network classifiers.
arXiv Detail & Related papers (2022-01-23T21:18:17Z) - Robust Neural Network Classification via Double Regularization [2.41710192205034]
We propose a novel double regularization of the neural network training loss that combines a penalty on the complexity of the classification model and an optimal reweighting of training observations.
We demonstrate DRFit, for neural net classification of (i) MNIST and (ii) CIFAR-10, in both cases with simulated mislabeling.
arXiv Detail & Related papers (2021-12-15T13:19:20Z) - The Interplay Between Implicit Bias and Benign Overfitting in Two-Layer
Linear Networks [51.1848572349154]
neural network models that perfectly fit noisy data can generalize well to unseen test data.
We consider interpolating two-layer linear neural networks trained with gradient flow on the squared loss and derive bounds on the excess risk.
arXiv Detail & Related papers (2021-08-25T22:01:01Z) - Attribute-Guided Adversarial Training for Robustness to Natural
Perturbations [64.35805267250682]
We propose an adversarial training approach which learns to generate new samples so as to maximize exposure of the classifier to the attributes-space.
Our approach enables deep neural networks to be robust against a wide range of naturally occurring perturbations.
arXiv Detail & Related papers (2020-12-03T10:17:30Z) - Adversarially Robust Learning via Entropic Regularization [31.6158163883893]
We propose a new family of algorithms, ATENT, for training adversarially robust deep neural networks.
Our approach achieves competitive (or better) performance in terms of robust classification accuracy.
arXiv Detail & Related papers (2020-08-27T18:54:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.