Robustness Emerges Early in Training Dynamics, but Is Not Preserved

• URL: http://arxiv.org/abs/2608.04442v1
• Date: Wed, 05 Aug 2026 04:34:17 GMT
• Title: Robustness Emerges Early in Training Dynamics, but Is Not Preserved
• Abstract summary: We propose a framework that performs strategic interventions on training dynamics to stabilize early-emergent robust priors.<n>Our approach includes two parameter-free strategies, which stabilize or recover robust shallow configurations without modifying the model architecture or introducing learnable parameters.
• Score: 5.69693035365895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robustness to natural corruptions remains a fundamental challenge for deep neural networks. In this paper, we identify a robustness fading phenomenon where shallow layers spontaneously develop robust representations and flat loss landscapes in early training, yet these properties are not preserved during standard convergence. To address this, we propose a framework that performs strategic interventions on training dynamics to stabilize the empirically identified early-emergent robust priors. Our approach includes two parameter-free strategies: Early-Phase Stabilization~(EPS) and Asymmetric Weight Reversion~(AWR), which stabilize or recover robust shallow configurations without modifying the model architecture or introducing learnable parameters. Extensive experiments demonstrate the efficacy of our framework across various benchmarks and architectures, yielding significant gains in downstream transfer, dynamic adaptation, and diverse computer vision applications.

Related papers

• Suppress and Diversify: Refining Robust Pathways for Corruption Robustness [5.2174037606370725]
  Suppress and Diversify (S&D) is a non-intrusive refinement approach that enhances robustness by dynamically selecting robust pathways.<n>S&D is architecture-agnostic, parameter-free, and incurs zero test-time overhead.
  arXiv  Detail & Related papers  (2026-08-07T02:11:59Z)
• TIDE: Asymmetric Neural Circuits for Stabilized Temporal Inhibitory-Excitatory Dynamics [51.37317469455798]
  Recent Continuous Thought Machine architecture decouples internal computation from external inputs via neural dynamics.<n>We propose to model neural dynamics using asymmetric Excitatory-Inhibitory (E-I) networks.<n>We introduce Temporal Inhibitory-Excitatory Dynamic Engine (TIDE), a neuro-inspired architecture that computes internal representations through neural dynamics stabilized.
  arXiv  Detail & Related papers  (2026-05-19T05:59:13Z)
• SECURE: Stable Early Collision Understanding via Robust Embeddings in Autonomous Driving [14.057807061501004]
  We introduce SECURE - Stable Early Collision Understanding Robust Embeddings, a framework that formally defines and enforces model robustness.<n>We propose a principled training methodology that fine-tunes a baseline model using a multi-objective loss, which minimizes divergence from a reference model and penalizes sensitivity to adversarial perturbations. Experiments on DAD and CCD datasets demonstrate that our approach not only significantly enhances robustness against various perturbations but also improves performance on clean data, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2026-04-01T19:42:32Z)
• Cross-Scenario Deraining Adaptation with Unpaired Data: Superpixel Structural Priors and Multi-Stage Pseudo-Rain Synthesis [48.809049043865905]
  Image deraining plays a pivotal role in low-level computer vision.<n>Deep learning paradigms often suffer from severe performance degradation when generalized to unseen Out-of-Distribution scenarios.<n>This paper proposes a pioneering cross-scenario deraining adaptation framework.
  arXiv  Detail & Related papers  (2026-03-23T07:38:55Z)
• If You Want to Be Robust, Be Wary of Initialization [28.195617869726636]
  Graph Neural Networks (GNNs) have demonstrated remarkable performance across a spectrum of graph-related tasks.<n>However, concerns persist regarding their vulnerability to adversarial perturbations.<n>We introduce a theoretical framework bridging the connection between initialization strategies and a network's resilience to adversarial perturbations.<n>Our analysis reveals a direct relationship between initial weights, number of training epochs and the model's vulnerability.
  arXiv  Detail & Related papers  (2025-10-26T12:28:12Z)
• Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning [58.533203990515034]
  Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL)<n>We show that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements.<n>We finally distill these insights into Module-Specific Training (MST), a practical framework that exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.
  arXiv  Detail & Related papers  (2025-10-14T03:03:08Z)
• Structured Transformations for Stable and Interpretable Neural Computation [1.2064681974642195]
  We introduce a reformulation of layer-level transformations that departs from the standard unconstrained affine paradigm.<n>Our formulation encourages internal consistency and supports stable information flow across depth.<n>We show that models constructed with these structured transformations exhibit improved gradient conditioning, reduced sensitivity to perturbations, and layer-wise robustness.
  arXiv  Detail & Related papers  (2025-07-31T19:26:45Z)
• Enhancing Variational Autoencoders with Smooth Robust Latent Encoding [54.74721202894622]
  Variational Autoencoders (VAEs) have played a key role in scaling up diffusion-based generative models.<n>We introduce Smooth Robust Latent VAE, a novel adversarial training framework that boosts both generation quality and robustness.<n>Experiments show that SRL-VAE improves both generation quality, in image reconstruction and text-guided image editing, and robustness, against Nightshade attacks and image editing attacks.
  arXiv  Detail & Related papers  (2025-04-24T03:17:57Z)
• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment.<n>We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos [5.854190253899593]
  In gradient descent dynamics of neural networks, the top eigenvalue of the loss Hessian (sharpness) displays a variety of robust phenomena throughout training.<n>We demonstrate that a simple $2$-layer linear network (UV model) trained on a single training example exhibits all of the essential sharpness phenomenology observed in real-world scenarios.
  arXiv  Detail & Related papers  (2023-11-03T17:59:40Z)
• Stabilizing Differentiable Architecture Search via Perturbation-based Regularization [99.81980366552408]
  We find that the precipitous validation loss landscape, which leads to a dramatic performance drop when distilling the final architecture, is an essential factor that causes instability. We propose a perturbation-based regularization - SmoothDARTS (SDARTS) - to smooth the loss landscape and improve the generalizability of DARTS-based methods.
  arXiv  Detail & Related papers  (2020-02-12T23:46:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.