Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks

• URL: http://arxiv.org/abs/2512.22192v1
• Date: Sat, 20 Dec 2025 11:33:32 GMT
• Title: Frequency Regularization: Unveiling the Spectral Inductive Bias of Deep Neural Networks
• Authors: Jiahao Lu,
• Abstract summary: We investigate the Spectral Bias of modern Convolutional Neural Networks (CNNs)<n>We introduce a Visual Diagnostic Framework to track the dynamic evolution of weight frequencies during training.<n>We propose a novel metric, the Spectral Suppression Ratio (SSR), to quantify the "low-pass filtering" intensity of different regularizers.
• Score: 21.749207209704906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Regularization techniques such as L2 regularization (Weight Decay) and Dropout are fundamental to training deep neural networks, yet their underlying physical mechanisms regarding feature frequency selection remain poorly understood. In this work, we investigate the Spectral Bias of modern Convolutional Neural Networks (CNNs). We introduce a Visual Diagnostic Framework to track the dynamic evolution of weight frequencies during training and propose a novel metric, the Spectral Suppression Ratio (SSR), to quantify the "low-pass filtering" intensity of different regularizers. By addressing the aliasing issue in small kernels (e.g., 3x3) through discrete radial profiling, our empirical results on ResNet-18 and CIFAR-10 demonstrate that L2 regularization suppresses high-frequency energy accumulation by over 3x compared to unregularized baselines. Furthermore, we reveal a critical Accuracy-Robustness Trade-off: while L2 models are sensitive to broadband Gaussian noise due to over-specialization in low frequencies, they exhibit superior robustness against high-frequency information loss (e.g., low resolution), outperforming baselines by >6% in blurred scenarios. This work provides a signal-processing perspective on generalization, confirming that regularization enforces a strong spectral inductive bias towards low-frequency structures.

Related papers

• Spectral Gating Networks [65.9496901693099]
  We introduce Spectral Gating Networks (SGN) to introduce frequency-rich expressivity in feed-forward networks.<n>SGN augments a standard activation pathway with a compact spectral pathway and learnable gates that allow the model to start from a stable base behavior.<n>It consistently improves accuracy-efficiency trade-offs under comparable computational budgets.
  arXiv  Detail & Related papers  (2026-02-07T20:00:49Z)
• The Homogeneity Trap: Spectral Collapse in Doubly-Stochastic Deep Networks [1.7523718031184992]
  We identify a critical spectral degradation phenomenon inherent to structure-preserving deep architectures.<n>We show that maximum-entropy bias drives the mixing operator towards the uniform barycenter, suppressing the subdominant singular value .<n>We derive a spectral bound linking to the network's effective depth, showing that high-entropy constraints restrict feature transformation to a shallow receptive field.
  arXiv  Detail & Related papers  (2026-01-05T13:09:42Z)
• Spectral Bottleneck in Deep Neural Networks: Noise is All You Need [0.0]
  We study the challenge of fitting high-frequency-dominant signals susceptible to spectral bottleneck.<n>To effectively fit any target signal irrespective of it's frequency content, we propose a generalized target perturbation scheme.<n>We show that the noise scales can provide control over the spectra of network activations and the eigenbasis of the empirical neural tangent kernel.
  arXiv  Detail & Related papers  (2025-09-09T22:16:24Z)
• SpectrumFM: Redefining Spectrum Cognition via Foundation Modeling [65.65474629224558]
  We propose a spectrum foundation model, termed SpectrumFM, which provides a new paradigm for spectrum cognition.<n>An innovative spectrum encoder that exploits the convolutional neural networks is proposed to effectively capture both fine-grained local signal structures and high-level global dependencies in the spectrum data.<n>Two novel self-supervised learning tasks, namely masked reconstruction and next-slot signal prediction, are developed for pre-training SpectrumFM, enabling the model to learn rich and transferable representations.
  arXiv  Detail & Related papers  (2025-08-02T14:40:50Z)
• SpINRv2: Implicit Neural Representation for Passband FMCW Radars [0.15193212081459279]
  We present SpINRv2, a neural framework for high-fidelity volumetric reconstruction using Frequency-Modulated Continuous-Wave radar.<n>Our core contribution is a fully differentiable frequency-domain forward model that captures the complex radar response using closed-form synthesis.<n>We introduce sparsity and regularization to disambiguate sub-bin ambiguities that arise at fine range resolutions.
  arXiv  Detail & Related papers  (2025-06-09T19:21:27Z)
• LOGLO-FNO: Efficient Learning of Local and Global Features in Fourier Neural Operators [20.77877474840923]
  High-frequency information is a critical challenge in machine learning.<n>Deep neural nets exhibit the so-called spectral bias toward learning low-frequency components.<n>We propose a novel frequency-sensitive loss term based on radially binned spectral errors.
  arXiv  Detail & Related papers  (2025-04-05T19:35:04Z)
• Mitigating Low-Frequency Bias: Feature Recalibration and Frequency Attention Regularization for Adversarial Robustness [23.77988226456179]
  adversarial training (AT) has emerged as a promising defense strategy.<n>AT-trained models exhibit a bias toward low-frequency features while neglecting high-frequency components.<n>We propose High-Frequency Feature Disentanglement and Recalibration (HFDR), a novel module that strategically separates and recalibrates frequency-specific features.
  arXiv  Detail & Related papers  (2024-07-04T15:46:01Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks [79.28094304325116]
  Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions. We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets. We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
  arXiv  Detail & Related papers  (2023-05-16T20:06:01Z)
• Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics [2.9443230571766854]
  We study the connection between the computations of ReLU networks, and the speed of gradient descent convergence. We then use this formulation to study the severity of spectral bias in low dimensional settings, and how positional encoding overcomes this.
  arXiv  Detail & Related papers  (2023-01-14T04:21:25Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• The Spectral Bias of Polynomial Neural Networks [63.27903166253743]
  Polynomial neural networks (PNNs) have been shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a $textitspectral bias$ towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Tangent Kernel (NTK) of PNNs. We find that the $Pi$-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the
  arXiv  Detail & Related papers  (2022-02-27T23:12:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.