Feature Dynamics as Implicit Data Augmentation: A Depth-Decomposed View on Deep Neural Network Generalization

• URL: http://arxiv.org/abs/2509.20334v2
• Date: Thu, 02 Oct 2025 20:09:02 GMT
• Title: Feature Dynamics as Implicit Data Augmentation: A Depth-Decomposed View on Deep Neural Network Generalization
• Authors: Tianyu Ruan, Kuo Gai, Shihua Zhang,
• Abstract summary: We show that temporal consistency extends to unseen and corrupted data, but collapses when semantic structure is destroyed.<n>Together, these findings suggest a conceptual perspective that links feature dynamics to generalization.
• Score: 18.72807692009739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Why do deep networks generalize well? In contrast to classical generalization theory, we approach this fundamental question by examining not only inputs and outputs, but the evolution of internal features. Our study suggests a phenomenon of temporal consistency that predictions remain stable when shallow features from earlier checkpoints combine with deeper features from later ones. This stability is not a trivial convergence artifact. It acts as a form of implicit, structured augmentation that supports generalization. We show that temporal consistency extends to unseen and corrupted data, but collapses when semantic structure is destroyed (e.g., random labels). Statistical tests further reveal that SGD injects anisotropic noise aligned with a few principal directions, reinforcing its role as a source of structured variability. Together, these findings suggest a conceptual perspective that links feature dynamics to generalization, pointing toward future work on practical surrogates for measuring temporal feature evolution.

Related papers

• Generalization Below the Edge of Stability: The Role of Data Geometry [60.147710896851045]
  We show how data geometry controls generalization in ReLU networks trained below the edge of stability.<n>For data distributions supported on a mixture of low-dimensional balls, we derive generalization bounds that provably adapt to the intrinsic dimension.<n>Our results consolidate disparate empirical findings that have appeared in the literature.
  arXiv  Detail & Related papers  (2025-10-20T21:40:36Z)
• Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility [90.894232610821]
  We analyze Transformers through the lens of rank structure.<n>We show that time-series embeddings exhibit sharply decaying singular value spectra.<n>We prove that the associated $Q/K/V$ projections admit accurate low-rank approximations.
  arXiv  Detail & Related papers  (2025-10-02T23:56:17Z)
• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting.<n>In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)
• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networks [3.566568169425391]
  We show that with increased depth, node representations become dominated by a low-dimensional subspace that depends on the aggregation function but not on the feature transformations. For all aggregation functions, the rank of the node representations collapses, resulting in over-smoothing for particular aggregation functions.
  arXiv  Detail & Related papers  (2023-08-31T15:22:31Z)
• DynGFN: Towards Bayesian Inference of Gene Regulatory Networks with GFlowNets [81.75973217676986]
  Gene regulatory networks (GRN) describe interactions between genes and their products that control gene expression and cellular function. Existing methods either focus on challenge (1), identifying cyclic structure from dynamics, or on challenge (2) learning complex Bayesian posteriors over DAGs, but not both. In this paper we leverage the fact that it is possible to estimate the "velocity" of gene expression with RNA velocity techniques to develop an approach that addresses both challenges.
  arXiv  Detail & Related papers  (2023-02-08T16:36:40Z)
• Isometric Representations in Neural Networks Improve Robustness [0.0]
  We train neural networks to perform classification while simultaneously maintaining within-class metric structure. We verify that isometric regularization improves the robustness to adversarial attacks on MNIST.
  arXiv  Detail & Related papers  (2022-11-02T16:18:18Z)
• Uniform Convergence, Adversarial Spheres and a Simple Remedy [40.44709296304123]
  Previous work has cast doubt on the general framework of uniform convergence and its ability to explain generalization in neural networks. We provide an extensive theoretical investigation of the previously studied data setting through the lens of infinitely-wide models. We prove that the Neural Tangent Kernel (NTK) also suffers from the same phenomenon and we uncover its origin.
  arXiv  Detail & Related papers  (2021-05-07T20:23:01Z)
• On Feature Decorrelation in Self-Supervised Learning [15.555208840500086]
  We study a framework containing the most common components from recent approaches. We connect dimensional collapse with strong correlations between axes and consider such connection as a strong motivation for feature decorrelation.
  arXiv  Detail & Related papers  (2021-05-02T13:28:18Z)
• Learning Causal Models Online [103.87959747047158]
  Predictive models can rely on spurious correlations in the data for making predictions. One solution for achieving strong generalization is to incorporate causal structures in the models. We propose an online algorithm that continually detects and removes spurious features.
  arXiv  Detail & Related papers  (2020-06-12T20:49:20Z)
• Understanding Generalization in Deep Learning via Tensor Methods [53.808840694241]
  We advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. We propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks.
  arXiv  Detail & Related papers  (2020-01-14T22:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.