A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights

• URL: http://arxiv.org/abs/2605.16913v1
• Date: Sat, 16 May 2026 10:03:41 GMT
• Title: A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights
• Abstract summary: We show that simple neural networks trained on image classification tasks first rely on amplitude information.<n>We then exploit phase information, which encodes edges and higher-order correlations.<n>We show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information.
• Score: 8.973965016201822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks trained with gradient-based methods exhibit a strong simplicity bias: they learn simpler statistical features of their data before moving to more complex features. Previous analyses of this phenomenon have largely focused on settings with (quasi-)isotropic inputs. In this work, we study the simplicity bias from a Fourier perspective, which allows us to include two key features of natural images in the analysis: approximate translation-invariance and power-law spectra. We first show experimentally that simple neural networks trained on image classification tasks first rely on amplitude information -- related to pair-wise correlations between pixels -- before exploiting phase information, which encodes edges and higher-order correlations. In view of this, we introduce a synthetic data model for translation-invariant inputs that allows precise control over amplitudes and phases while remaining tractable. We rigorously establish that for isotropic and high-dimensional inputs, classification based on phase information alone is a genuinely hard task: online stochastic gradient descent (SGD) cannot distinguish the structured inputs from noise within $n \ll N^3$ steps, but needs at least $n \gg N^3 \log^2{N}$ steps. In contrast, we show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information, even if the spectra do not help with classification. Simulations with two-layer networks trained on textures and with deep convolutional networks on ImageNet and CIFAR100 confirm this non-trivial interaction between amplitudes and phases, providing mechanistic insights into how deep neural networks can learn natural image distributions efficiently.

Related papers

• Self-Supervised Learning via Flow-Guided Neural Operator on Time-Series Data [57.85958428020496]
  Flow-Guided Neural Operator (FGNO) is a novel framework combining operator learning with flow matching for SSL training.<n>FGNO learns mappings in functional spaces by using Short-Time Fourier Transform to unify different time resolutions.<n>Unlike prior generative SSL methods that use noisy inputs during inference, we propose using clean inputs for representation extraction while learning representations with noise.
  arXiv  Detail & Related papers  (2026-02-12T18:54:57Z)
• How Learnable Grids Recover Fine Detail in Low Dimensions: A Neural Tangent Kernel Analysis of Multigrid Parametric Encodings [106.3726679697804]
  We compare the two most common techniques for mitigating this spectral bias: Fourier feature encodings (FFE) and multigrid parametric encodings (MPE)<n>MPEs are seen as the standard for low dimensional mappings, but MPEs often outperform them and learn representations with higher resolution and finer detail.<n>We prove that MPEs improve a network's performance through the structure of their grid and not their learnable embedding.
  arXiv  Detail & Related papers  (2025-04-18T02:18:08Z)
• Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions [7.2136602534376015]
  We investigate the interplay between data structure in feature learning for the most popular ICA algorithm, FastICA, and gradient descent (SGD)<n>We show that vanilla online SGD outperforms FastICA, and prove that the optimal sample complexity $ngtrsim d2$ can be reached by smoothing the loss.
  arXiv  Detail & Related papers  (2025-03-31T09:46:47Z)
• Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
  Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
  arXiv  Detail & Related papers  (2023-12-04T01:32:42Z)
• Coordinate-based Neural Network for Fourier Phase Retrieval [8.827173113748703]
  Single impliCit neurAl Network (SCAN) is a tool built upon coordinate neural networks meticulously designed for enhanced phase retrieval performance. SCAN adeptly connects object coordinates to their amplitude and phase within a unified network in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-11-25T04:23:23Z)
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks [79.28094304325116]
  Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions. We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets. We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
  arXiv  Detail & Related papers  (2023-05-16T20:06:01Z)
• Neural networks trained with SGD learn distributions of increasing complexity [78.30235086565388]
  We show that neural networks trained using gradient descent initially classify their inputs using lower-order input statistics. We then exploit higher-order statistics only later during training. We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of universality in learning.
  arXiv  Detail & Related papers  (2022-11-21T15:27:22Z)
• Spectral Evolution and Invariance in Linear-width Neural Networks [8.419660614226816]
  We investigate the spectral properties of linear-width feed-forward neural networks. We show that the spectra of weight in this high dimensional regime are invariant when trained by gradient descent for small constant learning rates. We also show that after adaptive gradient training, where a lower test error and feature learning emerge, both weight and kernel exhibit heavy tail behavior.
  arXiv  Detail & Related papers  (2022-11-11T23:00:30Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networks [43.860358308049044]
  In work, we show that these common perceptions can be completely false in the early phase of learning. We argue that this surprising simplicity can persist in networks with more layers with convolutional architecture.
  arXiv  Detail & Related papers  (2020-06-25T17:42:49Z)
• Extracting dispersion curves from ambient noise correlations using deep learning [1.0237120900821557]
  We present a machine-learning approach to classifying the phases of surface wave dispersion curves. Standard FTAN analysis of surfaces observed on an array of receivers is converted to an image. We use a convolutional neural network (U-net) architecture with a supervised learning objective and incorporate transfer learning.
  arXiv  Detail & Related papers  (2020-02-05T23:41:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.