How Learning Dynamics Drive Adversarially Robust Generalization?

• URL: http://arxiv.org/abs/2410.07719v2
• Date: Fri, 24 Oct 2025 07:21:32 GMT
• Title: How Learning Dynamics Drive Adversarially Robust Generalization?
• Authors: Yuelin Xu, Xiao Zhang,
• Abstract summary: We propose a novel PAC-Bayesian framework that explicitly links adversarial robustness to the posterior covariance of model parameters and the curvature of the adversarial loss landscape.<n>Our analyses reveal how key factors, such as learning rate, gradient noise, and Hessian structure, jointly shape robust generalization during training.
• Score: 3.7919737164481284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite significant progress in adversarially robust learning, the underlying mechanisms that govern robust generalization remain poorly understood. We propose a novel PAC-Bayesian framework that explicitly links adversarial robustness to the posterior covariance of model parameters and the curvature of the adversarial loss landscape. By characterizing discrete-time SGD dynamics near a local optimum under quadratic loss, we derive closed-form posterior covariances for both the stationary regime and the early phase of non-stationary transition. Our analyses reveal how key factors, such as learning rate, gradient noise, and Hessian structure, jointly shape robust generalization during training. Through empirical visualizations of these theoretical quantities, we fundamentally explain the phenomenon of robust overfitting and shed light on why flatness-promoting techniques like adversarial weight perturbation help to improve robustness.

Related papers

• Invariance on Manifolds: Understanding Robust Visual Representations for Place Recognition [19.200074425090595]
  We propose a Second-Order Geometric Statistics framework that inherently captures geometric stability without training.<n>Our approach introduces a training-free framework built upon fixed, pre-trained backbones, achieving strong zero-shot generalization without parameter updates.
  arXiv  Detail & Related papers  (2026-01-31T18:12:29Z)
• Bridging Information Asymmetry: A Hierarchical Framework for Deterministic Blind Face Restoration [31.878334664450776]
  We present textbfPrefRestore, a hierarchical framework that integrates discrete semantic logic with continuous texture generation.<n>Our methodology fundamentally addresses this information disparity through two complementary strategies.<n>Pref-Restore achieves state-of-the-art performance across synthetic and real-world benchmarks.
  arXiv  Detail & Related papers  (2026-01-27T11:50:31Z)
• Robust-R1: Degradation-Aware Reasoning for Robust Visual Understanding [54.05243949024302]
  Existing robust MLLMs rely on implicit training/adaptation that focuses solely on visual encoder generalization.<n>We propose Robust-R1, a novel framework that explicitly models visual degradations through structured reasoning chains.<n>Our approach integrates: (i) supervised fine-tuning for degradation-aware reasoning foundations, (ii) reward-driven alignment for accurately perceiving degradation parameters, and (iii) dynamic reasoning depth scaling adapted to degradation intensity.
  arXiv  Detail & Related papers  (2025-12-19T12:56:17Z)
• Adversarial Defence without Adversarial Defence: Enhancing Language Model Robustness via Instance-level Principal Component Removal [28.588188876688037]
  Pre-trained language models (PLMs) have driven substantial progress in natural language processing but remain vulnerable to adversarial attacks.<n>We propose a simple yet effective add-on module that enhances the adversarial robustness of PLMs.
  arXiv  Detail & Related papers  (2025-07-29T12:31:26Z)
• Preference Learning for AI Alignment: a Causal Perspective [55.2480439325792]
  We frame this problem in a causal paradigm, providing the rich toolbox of causality to identify persistent challenges.<n>Inheriting from the literature of causal inference, we identify key assumptions necessary for reliable generalisation.<n>We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness.
  arXiv  Detail & Related papers  (2025-06-06T10:45:42Z)
• On Weak-to-Strong Generalization and f-Divergence [23.062111583403095]
  Weak-to-strong generalization (W2SG) has emerged as a promising paradigm for stimulating the capabilities of strong pre-trained models.<n>We introduce $f$-divergence as an information-theoretic loss function framework in W2SG.<n>We empirically demonstrate that $f$-divergence loss, which generalizes widely-used metrics like KL divergence, effectively improves generalization and noise tolerance of the strong model in practice.
  arXiv  Detail & Related papers  (2025-06-03T17:40:08Z)
• A Flat Minima Perspective on Understanding Augmentations and Model Robustness [4.297070083645049]
  We offer a unified theoretical framework to clarify how augmentations can enhance model robustness.<n>Our work diverges from prior studies in that our analysis broadly encompasses much of the existing augmentation methods.<n>We confirm our theories through simulations on the existing common corruption and adversarial robustness benchmarks.
  arXiv  Detail & Related papers  (2025-05-30T13:40:44Z)
• Bridging Interpretability and Robustness Using LIME-Guided Model Refinement [0.0]
  Local Interpretable Model-Agnostic Explanations (LIME) systematically enhance model robustness. Empirical evaluations on multiple benchmark datasets demonstrate that LIME-guided refinement not only improves interpretability but also significantly enhances resistance to adversarial perturbations and generalization to out-of-distribution data.
  arXiv  Detail & Related papers  (2024-12-25T17:32:45Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Understanding Robust Overfitting from the Feature Generalization Perspective [61.770805867606796]
  Adversarial training (AT) constructs robust neural networks by incorporating adversarial perturbations into natural data. It is plagued by the issue of robust overfitting (RO), which severely damages the model's robustness. In this paper, we investigate RO from a novel feature generalization perspective.
  arXiv  Detail & Related papers  (2023-10-01T07:57:03Z)
• Doubly Robust Instance-Reweighted Adversarial Training [107.40683655362285]
  We propose a novel doubly-robust instance reweighted adversarial framework. Our importance weights are obtained by optimizing the KL-divergence regularized loss function. Our proposed approach outperforms related state-of-the-art baseline methods in terms of average robust performance.
  arXiv  Detail & Related papers  (2023-08-01T06:16:18Z)
• Sparsity-aware generalization theory for deep neural networks [12.525959293825318]
  We present a new approach to analyzing generalization for deep feed-forward ReLU networks. We show fundamental trade-offs between sparsity and generalization.
  arXiv  Detail & Related papers  (2023-07-01T20:59:05Z)
• Stochastic Modified Equations and Dynamics of Dropout Algorithm [4.811269936680572]
  Dropout is a widely utilized regularization technique in the training of neural networks. Its underlying mechanism and its impact on achieving good abilities remain poorly understood.
  arXiv  Detail & Related papers  (2023-05-25T08:42:25Z)
• On the generalization of learning algorithms that do not converge [54.122745736433856]
  Generalization analyses of deep learning typically assume that the training converges to a fixed point. Recent results indicate that in practice, the weights of deep neural networks optimized with gradient descent often oscillate indefinitely.
  arXiv  Detail & Related papers  (2022-08-16T21:22:34Z)
• Provable Generalization of Overparameterized Meta-learning Trained with SGD [62.892930625034374]
  We study the generalization of a widely used meta-learning approach, Model-Agnostic Meta-Learning (MAML) We provide both upper and lower bounds for the excess risk of MAML, which captures how SGD dynamics affect these generalization bounds. Our theoretical findings are further validated by experiments.
  arXiv  Detail & Related papers  (2022-06-18T07:22:57Z)
• Building Robust Ensembles via Margin Boosting [98.56381714748096]
  In adversarial robustness, a single model does not usually have enough power to defend against all possible adversarial attacks. We develop an algorithm for learning an ensemble with maximum margin. We show that our algorithm not only outperforms existing ensembling techniques, but also large models trained in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-06-07T14:55:58Z)
• Self-Ensemble Adversarial Training for Improved Robustness [14.244311026737666]
  Adversarial training is the strongest strategy against various adversarial attacks among all sorts of defense methods. Recent works mainly focus on developing new loss functions or regularizers, attempting to find the unique optimal point in the weight space. We devise a simple but powerful emphSelf-Ensemble Adversarial Training (SEAT) method for yielding a robust classifier by averaging weights of history models.
  arXiv  Detail & Related papers  (2022-03-18T01:12:18Z)
• The curse of overparametrization in adversarial training: Precise analysis of robust generalization for random features regression [34.35440701530876]
  We show that for adversarially trained random features models, high overparametrization can hurt robust generalization. Our developed theory reveals the nontrivial effect of overparametrization on robustness and indicates that for adversarially trained random features models, high overparametrization can hurt robust generalization.
  arXiv  Detail & Related papers  (2022-01-13T18:57:30Z)
• Towards Robust and Adaptive Motion Forecasting: A Causal Representation Perspective [72.55093886515824]
  We introduce a causal formalism of motion forecasting, which casts the problem as a dynamic process with three groups of latent variables. We devise a modular architecture that factorizes the representations of invariant mechanisms and style confounders to approximate a causal graph. Experiment results on synthetic and real datasets show that our three proposed components significantly improve the robustness and reusability of the learned motion representations.
  arXiv  Detail & Related papers  (2021-11-29T18:59:09Z)
• Exploring Transferable and Robust Adversarial Perturbation Generation from the Perspective of Network Hierarchy [52.153866313879924]
  The transferability and robustness of adversarial examples are two practical yet important properties for black-box adversarial attacks. We propose a transferable and robust adversarial generation (TRAP) method. Our TRAP achieves impressive transferability and high robustness against certain interferences.
  arXiv  Detail & Related papers  (2021-08-16T11:52:41Z)
• Formalizing Generalization and Robustness of Neural Networks to Weight Perturbations [58.731070632586594]
  We provide the first formal analysis for feed-forward neural networks with non-negative monotone activation functions against weight perturbations. We also design a new theory-driven loss function for training generalizable and robust neural networks against weight perturbations.
  arXiv  Detail & Related papers  (2021-03-03T06:17:03Z)
• Adversarially Robust Estimate and Risk Analysis in Linear Regression [17.931533943788335]
  Adversarially robust learning aims to design algorithms that are robust to small adversarial perturbations on input variables. By discovering the statistical minimax rate of convergence of adversarially robust estimators, we emphasize the importance of incorporating model information. We propose a straightforward two-stage adversarial learning framework, which facilitates to utilize model structure information to improve adversarial robustness.
  arXiv  Detail & Related papers  (2020-12-18T14:55:55Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• Can we have it all? On the Trade-off between Spatial and Adversarial Robustness of Neural Networks [21.664470275289403]
  We prove a quantitative trade-off between spatial and adversarial robustness in a simple statistical setting. We propose a method based on curriculum learning that trains gradually on more difficult perturbations (both spatial and adversarial) to improve spatial and adversarial robustness simultaneously.
  arXiv  Detail & Related papers  (2020-02-26T06:25:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.