Constructing Extreme Learning Machines with zero Spectral Bias

• URL: http://arxiv.org/abs/2307.09759v1
• Date: Wed, 19 Jul 2023 05:41:40 GMT
• Title: Constructing Extreme Learning Machines with zero Spectral Bias
• Abstract summary: Extreme Learning Machines (ELMs) obviate an iterative solution process which provides the theoretical basis of Spectral Bias (SB) This work verifies the reliability of this assumption, and shows that it is incorrect. The structure of ELMs makes them naturally amenable to implementation of variants of Fourier Feature Embeddings, which have been shown to mitigate SB in ANNs. This approach is implemented and verified to completely eliminate SB, thus bringing into feasibility the application of ELMs for practical problems like PINNs where resolution of higher frequencies is essential.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The phenomena of Spectral Bias, where the higher frequency components of a function being learnt in a feedforward Artificial Neural Network (ANN) are seen to converge more slowly than the lower frequencies, is observed ubiquitously across ANNs. This has created technology challenges in fields where resolution of higher frequencies is crucial, like in Physics Informed Neural Networks (PINNs). Extreme Learning Machines (ELMs) that obviate an iterative solution process which provides the theoretical basis of Spectral Bias (SB), should in principle be free of the same. This work verifies the reliability of this assumption, and shows that it is incorrect. However, the structure of ELMs makes them naturally amenable to implementation of variants of Fourier Feature Embeddings, which have been shown to mitigate SB in ANNs. This approach is implemented and verified to completely eliminate SB, thus bringing into feasibility the application of ELMs for practical problems like PINNs where resolution of higher frequencies is essential.

Related papers

• Frequency Selective Neural Networks as a Foundation Architecture for Time Series Learning [10.247004661473225]
  We introduce the Frequency Selective Neural Network (FSNN), pioneering a foundation architecture guaranteeing physical interpretability.<n>FSNN addresses spectral entanglement by explicitly embedding the rigorous mathematics of advanced signal processing into its neural topology.<n>Extensive evaluations demonstrate that FSNN establishes state-of-the-art predictive performance, achieving $77.0%$ average accuracy.
  arXiv  Detail & Related papers  (2026-08-29T02:51:29Z)
• When Does Frequency Decomposition Benefit Physics-Informed Neural Networks? A Preliminary Ablation Study [1.2081877372552605]
  Partial differential equations (PDEs) often have high-frequency and multi-scale features that neural networks struggle to approximate.<n>PINNs build the governing equations directly into training, but suffer from spectral bias.<n>We introduce a dual-branch, spectrally-gated architecture (DBSG-PINN) that splits low- and high-frequency components into separateworks joined by an adaptive gate.
  arXiv  Detail & Related papers  (2026-08-24T09:48:16Z)
• FreIE: Low-Frequency Spectral Bias in Neural Networks for Time-Series Tasks [5.215187965365735]
  We show that models tend to fit low-frequency signals before high-frequency ones.<n>We propose the FreLE algorithm, which enhances model generalization through both explicit and implicit frequency regularization.
  arXiv  Detail & Related papers  (2025-10-29T03:22:51Z)
• LOGLO-FNO: Efficient Learning of Local and Global Features in Fourier Neural Operators [20.77877474840923]
  High-frequency information is a critical challenge in machine learning. Deep neural nets exhibit the so-called spectral bias toward learning low-frequency components. We propose a novel frequency-sensitive loss term based on radially binned spectral errors.
  arXiv  Detail & Related papers  (2025-04-05T19:35:04Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks [79.28094304325116]
  Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions. We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets. We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
  arXiv  Detail & Related papers  (2023-05-16T20:06:01Z)
• Properties and Potential Applications of Random Functional-Linked Types of Neural Networks [81.56822938033119]
  Random functional-linked neural networks (RFLNNs) offer an alternative way of learning in deep structure. This paper gives some insights into the properties of RFLNNs from the viewpoints of frequency domain. We propose a method to generate a BLS network with better performance, and design an efficient algorithm for solving Poison's equation.
  arXiv  Detail & Related papers  (2023-04-03T13:25:22Z)
• Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics [2.9443230571766854]
  We study the connection between the computations of ReLU networks, and the speed of gradient descent convergence. We then use this formulation to study the severity of spectral bias in low dimensional settings, and how positional encoding overcomes this.
  arXiv  Detail & Related papers  (2023-01-14T04:21:25Z)
• Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders [0.0]
  An important inference from Neural Kernel Tangent (NTK) theory is the existence of spectral bias (SB) SB is low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. It is firmly established that under normalized conditions, PINNs do exhibit strong spectral bias, and this increases with the order of the differential equation.
  arXiv  Detail & Related papers  (2023-01-07T06:31:28Z)
• Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs [86.35471039808023]
  We introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases the number of frequency modes used by the model. We show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.
  arXiv  Detail & Related papers  (2022-11-28T09:57:15Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Understanding robustness and generalization of artificial neural networks through Fourier masks [8.94889125739046]
  Recent literature suggests that robust networks with good generalization properties tend to be biased towards processing low frequencies in images. We develop an algorithm that allows us to learn modulatory masks highlighting the essential input frequencies needed for preserving a trained network's performance.
  arXiv  Detail & Related papers  (2022-03-16T17:32:00Z)
• The Spectral Bias of Polynomial Neural Networks [63.27903166253743]
  Polynomial neural networks (PNNs) have been shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a $textitspectral bias$ towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Tangent Kernel (NTK) of PNNs. We find that the $Pi$-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the
  arXiv  Detail & Related papers  (2022-02-27T23:12:43Z)
• On the exact computation of linear frequency principle dynamics and its generalization [6.380166265263755]
  Recent works show an intriguing phenomenon of Frequency Principle (F-Principle) that fits the target function from low to high frequency during the training. In this paper, we derive the exact differential equation, namely Linear Frequency-Principle (LFP) model, governing the evolution of NN output function in frequency domain.
  arXiv  Detail & Related papers  (2020-10-15T15:17:21Z)
• Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks [9.23835409289015]
  We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -- DNNs often fit target functions from low to high frequencies.
  arXiv  Detail & Related papers  (2019-01-19T13:37:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.