On the study of frequency control and spectral bias in Wavelet-Based Kolmogorov Arnold networks: A path to physics-informed KANs

• URL: http://arxiv.org/abs/2502.00280v1
• Date: Sat, 01 Feb 2025 02:35:12 GMT
• Title: On the study of frequency control and spectral bias in Wavelet-Based Kolmogorov Arnold networks: A path to physics-informed KANs
• Authors: Juan Daniel Meshir, Abel Palafox, Edgar Alejandro Guerrero,
• Abstract summary: Spectral bias, the tendency of neural networks to prioritize learning low-frequency components of functions during the initial training stages, poses a significant challenge when approximating solutions with high-frequency details. We analyze the eigenvalues of the neural tangent kernel (NTK) of Wavelet Kolmogorov Arnold Networks (Wav-KANs) to enhance their ability to converge on high-frequency components.
• Score: 0.35998666903987897
• License:
• Abstract: Spectral bias, the tendency of neural networks to prioritize learning low-frequency components of functions during the initial training stages, poses a significant challenge when approximating solutions with high-frequency details. This issue is particularly pronounced in physics-informed neural networks (PINNs), widely used to solve differential equations that describe physical phenomena. In the literature, contributions such as Wavelet Kolmogorov Arnold Networks (Wav-KANs) have demonstrated promising results in capturing both low- and high-frequency components. Similarly, Fourier features (FF) are often employed to address this challenge. However, the theoretical foundations of Wav-KANs, particularly the relationship between the frequency of the mother wavelet and spectral bias, remain underexplored. A more in-depth understanding of how Wav-KANs manage high-frequency terms could offer valuable insights for addressing oscillatory phenomena encountered in parabolic, elliptic, and hyperbolic differential equations. In this work, we analyze the eigenvalues of the neural tangent kernel (NTK) of Wav-KANs to enhance their ability to converge on high-frequency components, effectively mitigating spectral bias. Our theoretical findings are validated through numerical experiments, where we also discuss the limitations of traditional approaches, such as standard PINNs and Fourier features, in addressing multi-frequency problems.

Related papers

• Robustifying Fourier Features Embeddings for Implicit Neural Representations [25.725097757343367]
  Implicit Neural Representations (INRs) employ neural networks to represent continuous functions by mapping coordinates to the corresponding values of the target function. INRs face a challenge known as spectral bias when dealing with scenes containing varying frequencies. We propose the use of multi-layer perceptrons (MLPs) without additive.
  arXiv  Detail & Related papers  (2025-02-08T07:43:37Z)
• Quantum error mitigation for Fourier moment computation [49.1574468325115]
  This paper focuses on the computation of Fourier moments within the context of a nuclear effective field theory on superconducting quantum hardware. The study integrates echo verification and noise renormalization into Hadamard tests using control reversal gates. The analysis, conducted using noise models, reveals a significant reduction in noise strength by two orders of magnitude.
  arXiv  Detail & Related papers  (2024-01-23T19:10:24Z)
• A Scalable Walsh-Hadamard Regularizer to Overcome the Low-degree Spectral Bias of Neural Networks [79.28094304325116]
  Despite the capacity of neural nets to learn arbitrary functions, models trained through gradient descent often exhibit a bias towards simpler'' functions. We show how this spectral bias towards low-degree frequencies can in fact hurt the neural network's generalization on real-world datasets. We propose a new scalable functional regularization scheme that aids the neural network to learn higher degree frequencies.
  arXiv  Detail & Related papers  (2023-05-16T20:06:01Z)
• Understanding the Spectral Bias of Coordinate Based MLPs Via Training Dynamics [2.9443230571766854]
  We study the connection between the computations of ReLU networks, and the speed of gradient descent convergence. We then use this formulation to study the severity of spectral bias in low dimensional settings, and how positional encoding overcomes this.
  arXiv  Detail & Related papers  (2023-01-14T04:21:25Z)
• Investigations on convergence behaviour of Physics Informed Neural Networks across spectral ranges and derivative orders [0.0]
  An important inference from Neural Kernel Tangent (NTK) theory is the existence of spectral bias (SB) SB is low frequency components of the target function of a fully connected Artificial Neural Network (ANN) being learnt significantly faster than the higher frequencies during training. This is established for Mean Square Error (MSE) loss functions with very low learning rate parameters. It is firmly established that under normalized conditions, PINNs do exhibit strong spectral bias, and this increases with the order of the differential equation.
  arXiv  Detail & Related papers  (2023-01-07T06:31:28Z)
• Incremental Spatial and Spectral Learning of Neural Operators for Solving Large-Scale PDEs [86.35471039808023]
  We introduce the Incremental Fourier Neural Operator (iFNO), which progressively increases the number of frequency modes used by the model. We show that iFNO reduces total training time while maintaining or improving generalization performance across various datasets. Our method demonstrates a 10% lower testing error, using 20% fewer frequency modes compared to the existing Fourier Neural Operator, while also achieving a 30% faster training.
  arXiv  Detail & Related papers  (2022-11-28T09:57:15Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• The Spectral Bias of Polynomial Neural Networks [63.27903166253743]
  Polynomial neural networks (PNNs) have been shown to be particularly effective at image generation and face recognition, where high-frequency information is critical. Previous studies have revealed that neural networks demonstrate a $textitspectral bias$ towards low-frequency functions, which yields faster learning of low-frequency components during training. Inspired by such studies, we conduct a spectral analysis of the Tangent Kernel (NTK) of PNNs. We find that the $Pi$-Net family, i.e., a recently proposed parametrization of PNNs, speeds up the
  arXiv  Detail & Related papers  (2022-02-27T23:12:43Z)
• Overview frequency principle/spectral bias in deep learning [8.78791231619729]
  We show a Frequency Principle (F-Principle) of the training behavior of deep neural networks (DNNs) The F-Principle is first demonstrated by onedimensional synthetic data followed by the verification in high-dimensional real datasets. This low-frequency implicit bias reveals the strength of neural network in learning low-frequency functions as well as its deficiency in learning high-frequency functions.
  arXiv  Detail & Related papers  (2022-01-19T03:08:33Z)
• F-FADE: Frequency Factorization for Anomaly Detection in Edge Streams [53.70940420595329]
  We propose F-FADE, a new approach for detection of anomalies in edge streams. It uses a novel frequency-factorization technique to efficiently model the time-evolving distributions of frequencies of interactions between node-pairs. F-FADE is able to handle in an online streaming setting a broad variety of anomalies with temporal and structural changes, while requiring only constant memory.
  arXiv  Detail & Related papers  (2020-11-09T19:55:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.