The Geometry of Adversarial Training in Binary Classification

• URL: http://arxiv.org/abs/2111.13613v1
• Date: Fri, 26 Nov 2021 17:19:50 GMT
• Title: The Geometry of Adversarial Training in Binary Classification
• Authors: Leon Bungert, Nicol\'as Garc\'ia Trillos, Ryan Murray
• Abstract summary: We establish an equivalence between a family of adversarial training problems for non-parametric binary classification and a family of regularized risk minimization problems. The resulting regularized risk minimization problems admit exact convex relaxations of the type $L1+$ (nonlocal) $operatornameTV$.
• Score: 1.2891210250935143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We establish an equivalence between a family of adversarial training problems for non-parametric binary classification and a family of regularized risk minimization problems where the regularizer is a nonlocal perimeter functional. The resulting regularized risk minimization problems admit exact convex relaxations of the type $L^1+$ (nonlocal) $\operatorname{TV}$, a form frequently studied in image analysis and graph-based learning. A rich geometric structure is revealed by this reformulation which in turn allows us to establish a series of properties of optimal solutions of the original problem, including the existence of minimal and maximal solutions (interpreted in a suitable sense), and the existence of regular solutions (also interpreted in a suitable sense). In addition, we highlight how the connection between adversarial training and perimeter minimization problems provides a novel, directly interpretable, statistical motivation for a family of regularized risk minimization problems involving perimeter/total variation. The majority of our theoretical results are independent of the distance used to define adversarial attacks.

Related papers

• On the Geometry of Regularization in Adversarial Training: High-Dimensional Asymptotics and Generalization Bounds [11.30047438005394]
  This work investigates the question of how to choose the regularization norm $lVert cdot rVert$ in the context of high-dimensional adversarial training for binary classification. We quantitatively characterize the relationship between perturbation size and the optimal choice of $lVert cdot rVert$, confirming the intuition that, in the data scarce regime, the type of regularization becomes increasingly important for adversarial training as perturbations grow in size.
  arXiv  Detail & Related papers  (2024-10-21T14:53:12Z)
• A mean curvature flow arising in adversarial training [1.2289361708127877]
  We connect adversarial training for binary classification to a geometric evolution equation for the decision boundary. We prove that the scheme is monotone and consistent as the adversarial budget vanishes and the perimeter localizes. This highlights that the efficacy of adversarial training may be due to locally minimizing the length of the decision boundary.
  arXiv  Detail & Related papers  (2024-04-22T17:58:36Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Riemannian stochastic optimization methods avoid strict saddle points [68.80251170757647]
  We show that policies under study avoid strict saddle points / submanifolds with probability 1. This result provides an important sanity check as it shows that, almost always, the limit state of an algorithm can only be a local minimizer.
  arXiv  Detail & Related papers  (2023-11-04T11:12:24Z)
• Specifying and Solving Robust Empirical Risk Minimization Problems Using CVXPY [39.47096356027248]
  In some simple cases, such problems can be expressed in an analytical form. In general the problem can be made tractable via dualization, which turns a min-max problem into a min-min problem. We demonstrate how CVXPY can be used to automate this dualization procedure in a user-friendly manner.
  arXiv  Detail & Related papers  (2023-06-09T03:35:33Z)
• It begins with a boundary: A geometric view on probabilistically robust learning [6.877576704011329]
  We take a fresh and geometric view on one such method -- Probabilistically Robust Learning (PRL) We prove existence of solutions to the original and modified problems using novel relaxation methods. We also clarify, through a suitable $Gamma$-convergence analysis, the way in which the original and modified PRL models interpolate between risk minimization and adversarial training.
  arXiv  Detail & Related papers  (2023-05-30T06:24:30Z)
• Mitigating multiple descents: A model-agnostic framework for risk monotonization [84.6382406922369]
  We develop a general framework for risk monotonization based on cross-validation. We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
  arXiv  Detail & Related papers  (2022-05-25T17:41:40Z)
• Optimal variance-reduced stochastic approximation in Banach spaces [114.8734960258221]
  We study the problem of estimating the fixed point of a contractive operator defined on a separable Banach space. We establish non-asymptotic bounds for both the operator defect and the estimation error.
  arXiv  Detail & Related papers  (2022-01-21T02:46:57Z)
• A Theorem of the Alternative for Personalized Federated Learning [19.499120576896228]
  We show how excess risks of personalized federated learning depend on data heterogeneity from a minimax point of view. Our results show that the presumably difficult (infinite-dimensional) problem of adapting to client-wise heterogeneity can be reduced to a simple binary decision problem.
  arXiv  Detail & Related papers  (2021-03-02T17:58:20Z)
• Fundamental Limits and Tradeoffs in Invariant Representation Learning [99.2368462915979]
  Many machine learning applications involve learning representations that achieve two competing goals. Minimax game-theoretic formulation represents a fundamental tradeoff between accuracy and invariance. We provide an information-theoretic analysis of this general and important problem under both classification and regression settings.
  arXiv  Detail & Related papers  (2020-12-19T15:24:04Z)
• Total Deep Variation: A Stable Regularizer for Inverse Problems [71.90933869570914]
  We introduce the data-driven general-purpose total deep variation regularizer. In its core, a convolutional neural network extracts local features on multiple scales and in successive blocks. We achieve state-of-the-art results for numerous imaging tasks.
  arXiv  Detail & Related papers  (2020-06-15T21:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.