RepNN: Tackling spectral bias in deep neural networks via parameter reparameterization

• URL: http://arxiv.org/abs/2606.16575v2
• Date: Thu, 18 Jun 2026 14:55:17 GMT
• Title: RepNN: Tackling spectral bias in deep neural networks via parameter reparameterization
• Abstract summary: Deep neural networks (DNNs) have achieved remarkable success in scientific computing.<n>They often suffer from spectral bias in capturing oscillatory and multiscale behaviors.<n>In this study, we investigate the failure of shallow ReLU neural networks in fitting high-frequency functions.
• Score: 8.528546373000838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNN, a reparameterized neural network model with activation ReLU or tanh designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one- and four-dimensional function approximations, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning for an earthquake problem using real data, demonstrate that RepNN improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features with slightly additional computational cost. These results indicate that RepNN provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.

Related papers

• SBS: Enhancing Parameter-Efficiency of Neural Representations for Neural Networks via Spectral Bias Suppression [6.410718573605]
  Implicit neural representations have been extended to represent convolutional neural network weights via neural representation for neural networks.<n>Standard multi-layer perceptrons used in neural representation for neural networks exhibit a pronounced spectral bias.<n>We propose SBS, a parameter-efficient enhancement to neural representation for neural networks that suppresses spectral bias.
  arXiv  Detail & Related papers  (2025-09-09T03:48:57Z)
• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval.<n>A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed.<n>The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Frequency-adaptive Multi-scale Deep Neural Networks [6.338572283139395]
  We propose frequency-adaptive MscaleDNNs for approximating high frequency functions. These MscaleDNNs improve accuracy by two to three orders of magnitude compared to standard MscaleDNNs.
  arXiv  Detail & Related papers  (2024-09-28T14:49:23Z)
• Residual resampling-based physics-informed neural network for neutron diffusion equations [7.105073499157097]
  The neutron diffusion equation plays a pivotal role in the analysis of nuclear reactors. Traditional PINN approaches often utilize fully connected network (FCN) architecture. R2-PINN effectively overcomes the limitations inherent in current methods, providing more accurate and robust solutions for neutron diffusion equations.
  arXiv  Detail & Related papers  (2024-06-23T13:49:31Z)
• Deeper or Wider: A Perspective from Optimal Generalization Error with Sobolev Loss [11.342593850148758]
  We compare deeper neural networks (DeNNs) with a flexible number of layers and wider neural networks (WeNNs) with limited hidden layers.<n>We find that a higher number of parameters tends to favor WeNNs, while an increased number of sample points and greater regularity in the loss function lean towards the adoption of DeNNs.
  arXiv  Detail & Related papers  (2024-01-31T20:10:10Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration [21.117214351356765]
  Spiking Neural Network (SNN) is recognized as one of the next-generation neural networks. In this work, we argue that simply copying and pasting the weights of ANN to SNN inevitably results in activation mismatch. We propose a set of layer-wise parameter calibration algorithms, which adjusts the parameters to minimize the activation mismatch.
  arXiv  Detail & Related papers  (2022-05-06T18:22:09Z)
• LocalDrop: A Hybrid Regularization for Deep Neural Networks [98.30782118441158]
  We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs) has been developed based on the proposed upper bound of the local Rademacher complexity.
  arXiv  Detail & Related papers  (2021-03-01T03:10:11Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks [107.77595511218429]
  In this paper, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks. We propose a feature distortion method (Disout) for addressing the aforementioned problem. The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated.
  arXiv  Detail & Related papers  (2020-02-23T13:59:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.