Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Case

• URL: http://arxiv.org/abs/2606.21488v1
• Date: Fri, 19 Jun 2026 14:38:01 GMT
• Title: Robustness Cannot be Reduced to Regularization: Studying Adversarial Training Beyond the Linear Case
• Abstract summary: adversarial training is one of the most effective countermeasures to this issue.<n>High computational cost remains an obstacle to practical deployment.<n>Recent progress in reducing this cost has relied on a formal equivalence between the adversarial risk and a simpler form of regularized risk.
• Score: 11.550543488697018
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The vulnerability of ML models to adversarial examples has recently emerged as a major concern. While adversarial training is one of the most effective countermeasures to this issue, its high computational cost remains an obstacle to practical deployment. Recent progress in reducing this cost has relied, in the case of linear models, on a formal equivalence between the adversarial risk and a simpler form of regularized risk. This enabled significantly more efficient training procedures, which naturally raises the question of whether such an equivalence can be extended beyond linear models. In this work, we formally show that no such equivalence is possible for two-layer networks. Our proofs proceed via a reduction to key properties that fundamentally separate the adversarial risk from any simple regularized risk which would only exhibit a weak form of data dependence. Beyond this setting, we provide empirical evidence on Wide-ResNets indicating that the same type of impossibility persists in deeper and more expressive architectures.

Related papers

• Expected Harm: Rethinking Safety Evaluation of (Mis)Aligned LLMs [24.88278177777809]
  We introduce Expected Harm, a metric that weights the severity of a jailbreak by its execution likelihood, modeled as a function of execution cost.<n>We demonstrate that this miscalibration creates a structural vulnerability: by exploiting this property, we increase the attack success rate of existing jailbreaks by up to $2times$.
  arXiv  Detail & Related papers  (2026-02-02T03:48:04Z)
• Backdoor Unlearning by Linear Task Decomposition [69.91984435094157]
  Foundation models are highly susceptible to adversarial perturbations and targeted backdoor attacks.<n>Existing backdoor removal approaches rely on costly fine-tuning to override the harmful behavior.<n>This raises the question of whether backdoors can be removed without compromising the general capabilities of the models.
  arXiv  Detail & Related papers  (2025-10-16T16:18:07Z)
• AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models [62.70575022567081]
  We propose AdvChain, an alignment paradigm that teaches models dynamic self-correction through adversarial CoT tuning.<n>Our work establishes a new direction for building more robust and reliable reasoning models.
  arXiv  Detail & Related papers  (2025-09-29T04:27:23Z)
• Achievable distributional robustness when the robust risk is only partially identified [8.192907805418583]
  In safety-critical applications, machine learning models should generalize well under worst-case distribution shifts.<n>We introduce the worst-case robust risk as a new measure of robustness that is always well-defined regardless of identifiability.<n>First, we show that existing robustness methods are provably suboptimal in the partially identifiable case.
  arXiv  Detail & Related papers  (2025-02-04T20:42:47Z)
• Unlearning Backdoor Threats: Enhancing Backdoor Defense in Multimodal Contrastive Learning via Local Token Unlearning [49.242828934501986]
  Multimodal contrastive learning has emerged as a powerful paradigm for building high-quality features. backdoor attacks subtly embed malicious behaviors within the model during training. We introduce an innovative token-based localized forgetting training regime.
  arXiv  Detail & Related papers  (2024-03-24T18:33:15Z)
• Disparate Impact on Group Accuracy of Linearization for Private Inference [48.27026603581436]
  We show that reducing the number of ReLU activations disproportionately decreases the accuracy for minority groups compared to majority groups. We also show how a simple procedure altering the fine-tuning step for linearized models can serve as an effective mitigation strategy.
  arXiv  Detail & Related papers  (2024-02-06T01:56:29Z)
• The Surprising Harmfulness of Benign Overfitting for Adversarial Robustness [13.120373493503772]
  We prove a surprising result that even if the ground truth itself is robust to adversarial examples, the benignly overfitted model is benign in terms of the standard'' out-of-sample risk objective. Our finding provides theoretical insights into the puzzling phenomenon observed in practice, where the true target function (e.g., human) is robust against adverasrial attack, while beginly overfitted neural networks lead to models that are not robust.
  arXiv  Detail & Related papers  (2024-01-19T15:40:46Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• Seeing is not Believing: Robust Reinforcement Learning against Spurious Correlation [57.351098530477124]
  We consider one critical type of robustness against spurious correlation, where different portions of the state do not have correlations induced by unobserved confounders. A model that learns such useless or even harmful correlation could catastrophically fail when the confounder in the test case deviates from the training one. Existing robust algorithms that assume simple and unstructured uncertainty sets are therefore inadequate to address this challenge.
  arXiv  Detail & Related papers  (2023-07-15T23:53:37Z)
• Enhancing Multiple Reliability Measures via Nuisance-extended Information Bottleneck [77.37409441129995]
  In practical scenarios where training data is limited, many predictive signals in the data can be rather from some biases in data acquisition. We consider an adversarial threat model under a mutual information constraint to cover a wider class of perturbations in training. We propose an autoencoder-based training to implement the objective, as well as practical encoder designs to facilitate the proposed hybrid discriminative-generative training.
  arXiv  Detail & Related papers  (2023-03-24T16:03:21Z)
• Overparameterized Linear Regression under Adversarial Attacks [0.0]
  We study the error of linear regression in the face of adversarial attacks. We show that adding features to linear models might be either a source of additional robustness or brittleness.
  arXiv  Detail & Related papers  (2022-04-13T09:50:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.