Spectral Bottleneck in Deep Neural Networks: Noise is All You Need

• URL: http://arxiv.org/abs/2509.09719v1
• Date: Tue, 09 Sep 2025 22:16:24 GMT
• Title: Spectral Bottleneck in Deep Neural Networks: Noise is All You Need
• Authors: Hemanth Chandravamsi, Dhanush V. Shenoy, Itay Zinn, Shimon Pisnoy, Steven H. Frankel,
• Abstract summary: We study the challenge of fitting high-frequency-dominant signals susceptible to spectral bottleneck.<n>To effectively fit any target signal irrespective of it's frequency content, we propose a generalized target perturbation scheme.<n>We show that the noise scales can provide control over the spectra of network activations and the eigenbasis of the empirical neural tangent kernel.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks are known to exhibit a spectral learning bias, wherein low-frequency components are learned early in training, while high-frequency modes emerge more gradually in later epochs. However, when the target signal lacks low-frequency components and is dominated by broadband high frequencies, training suffers from a 'spectral bottleneck', and the model fails to reconstruct the entire signal, including the frequency components that lie within the network's representational capacity. We examine such a scenario in the context of implicit neural representations (INRs) with sinusoidal representation networks (SIRENs), focusing on the challenge of fitting high-frequency-dominant signals that are susceptible to spectral bottleneck. To effectively fit any target signal irrespective of it's frequency content, we propose a generalized target-aware 'weight perturbation scheme' (WINNER - weight initialization with noise for neural representations) for network initialization. The scheme perturbs uniformly initialized weights with Gaussian noise, where the noise scales are adaptively determined by the spectral centroid of the target signal. We show that the noise scales can provide control over the spectra of network activations and the eigenbasis of the empirical neural tangent kernel. This method not only addresses the spectral bottleneck but also yields faster convergence and with improved representation accuracy, outperforming state-of-the-art approaches in audio fitting and achieving notable gains in image fitting and denoising tasks. Beyond signal reconstruction, our approach opens new directions for adaptive weight initialization strategies in computer vision and scientific machine learning.

Related papers

• A Unified Frequency Principle for Quantum and Classical Machine Learning [9.529771617722703]
  We present a unified theoretical framework for the frequency principle (F-principle) that characterizes the training dynamics of quantum neural networks.<n>Within this framework, we prove that quantum neural networks exhibit a spectral bias toward learning low-frequency components of target functions.<n>Our results provide a frequency-domain lens that unifies classical and quantum learning dynamics, clarifies the role of noise in shaping trainability, and guides the design of noise-resilient quantum neural networks.
  arXiv  Detail & Related papers  (2026-01-06T16:44:22Z)
• Neural network initialization with nonlinear characteristics and information on spectral bias [0.0]
  Initialization of neural network parameters, such as weights and biases, has a crucial impact on learning performance.<n>We propose a framework that adjusts the scale factors in the SWIM algorithm to capture low-frequency components in the early-stage hidden layers.
  arXiv  Detail & Related papers  (2025-11-04T04:15:32Z)
• Improving Accuracy and Efficiency of Implicit Neural Representations: Making SIREN a WINNER [0.0]
  We identify and address a fundamental limitation of sinusoidal representation networks (SIRENs)<n>In extreme cases, when the network's frequency support misaligns with the target spectrum, a'spectral bottleneck' is observed.<n>We propose WINNER - Weight Initialization with Noise for Neural Representations.
  arXiv  Detail & Related papers  (2025-09-16T11:41:13Z)
• Neuromorphic Wireless Split Computing with Resonate-and-Fire Neurons [69.73249913506042]
  This paper investigates a wireless split computing architecture that employs resonate-and-fire (RF) neurons to process time-domain signals directly.<n>By resonating at tunable frequencies, RF neurons extract time-localized spectral features while maintaining low spiking activity.<n> Experimental results show that the proposed RF-SNN architecture achieves comparable accuracy to conventional LIF-SNNs and ANNs.
  arXiv  Detail & Related papers  (2025-06-24T21:14:59Z)
• Scintillation pulse characterization with spectrum-inspired temporal neural networks: case studies on particle detector signals [1.124958340749622]
  We propose a network architecture specially tailored for scintillation pulse characterization based on previous works on time series analysis.<n>We prove our idea in two case studies: (a) simulation data generated with the setting of the LUX dark matter detector, and (b) experimental electrical signals with fast electronics to emulate scintillation variations for the NICA/MPD calorimeter.
  arXiv  Detail & Related papers  (2024-10-09T02:44:53Z)
• Tuning the Frequencies: Robust Training for Sinusoidal Neural Networks [1.5124439914522694]
  We introduce a theoretical framework that explains the capacity property of sinusoidal networks.<n>We show how its layer compositions produce a large number of new frequencies expressed as integer combinations of the input frequencies.<n>Our method, referred to as TUNER, greatly improves the stability and convergence of sinusoidal INR training, leading to detailed reconstructions.
  arXiv  Detail & Related papers  (2024-07-30T18:24:46Z)
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks [79.28094304325116]
  Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions. We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets. We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
  arXiv  Detail & Related papers  (2023-05-16T20:06:01Z)
• Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics [2.9443230571766854]
  We study the connection between the computations of ReLU networks, and the speed of gradient descent convergence. We then use this formulation to study the severity of spectral bias in low dimensional settings, and how positional encoding overcomes this.
  arXiv  Detail & Related papers  (2023-01-14T04:21:25Z)
• NAF: Neural Attenuation Fields for Sparse-View CBCT Reconstruction [79.13750275141139]
  This paper proposes a novel and fast self-supervised solution for sparse-view CBCT reconstruction. The desired attenuation coefficients are represented as a continuous function of 3D spatial coordinates, parameterized by a fully-connected deep neural network. A learning-based encoder entailing hash coding is adopted to help the network capture high-frequency details.
  arXiv  Detail & Related papers  (2022-09-29T04:06:00Z)
• Meta-Learning Sparse Implicit Neural Representations [69.15490627853629]
  Implicit neural representations are a promising new avenue of representing general signals. Current approach is difficult to scale for a large number of signals or a data set. We show that meta-learned sparse neural representations achieve a much smaller loss than dense meta-learned models.
  arXiv  Detail & Related papers  (2021-10-27T18:02:53Z)
• SignalNet: A Low Resolution Sinusoid Decomposition and Estimation Network [79.04274563889548]
  We propose SignalNet, a neural network architecture that detects the number of sinusoids and estimates their parameters from quantized in-phase and quadrature samples. We introduce a worst-case learning threshold for comparing the results of our network relative to the underlying data distributions. In simulation, we find that our algorithm is always able to surpass the threshold for three-bit data but often cannot exceed the threshold for one-bit data.
  arXiv  Detail & Related papers  (2021-06-10T04:21:20Z)
• Deep Networks for Direction-of-Arrival Estimation in Low SNR [89.45026632977456]
  We introduce a Convolutional Neural Network (CNN) that is trained from mutli-channel data of the true array manifold matrix. We train a CNN in the low-SNR regime to predict DoAs across all SNRs. Our robust solution can be applied in several fields, ranging from wireless array sensors to acoustic microphones or sonars.
  arXiv  Detail & Related papers  (2020-11-17T12:52:18Z)
• Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization [26.270754571140735]
  PIM accelerators often suffer from intrinsic noise in the physical components. We propose a noise-agnostic method to achieve robust neural network performance against any noise setting.
  arXiv  Detail & Related papers  (2020-07-07T06:51:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.