Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations

• URL: http://arxiv.org/abs/2411.15111v4
• Date: Fri, 13 Jun 2025 07:52:33 GMT
• Title: Learnable Activation Functions in Physics-Informed Neural Networks for Solving Partial Differential Equations
• Authors: Afrah Farea, Mustafa Serdar Celebi,
• Abstract summary: Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs)<n>These limitations impact their accuracy for problems involving rapid oscillations, sharp gradients, and complex boundary behaviors.<n>We investigate learnable activation functions as a solution to these challenges.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs). However, they face challenges related to spectral bias (the tendency to learn low-frequency components while struggling with high-frequency features) and unstable convergence dynamics (mainly stemming from the multi-objective nature of the PINN loss function). These limitations impact their accuracy for problems involving rapid oscillations, sharp gradients, and complex boundary behaviors. We systematically investigate learnable activation functions as a solution to these challenges, comparing Multilayer Perceptrons (MLPs) using fixed and learnable activation functions against Kolmogorov-Arnold Networks (KANs) that employ learnable basis functions. Our evaluation spans diverse PDE types, including linear and non-linear wave problems, mixed-physics systems, and fluid dynamics. Using empirical Neural Tangent Kernel (NTK) analysis and Hessian eigenvalue decomposition, we assess spectral bias and convergence stability of the models. Our results reveal a trade-off between expressivity and training convergence stability. While learnable activation functions work well in simpler architectures, they encounter scalability issues in complex networks due to the higher functional dimensionality. Counterintuitively, we find that low spectral bias alone does not guarantee better accuracy, as functions with broader NTK eigenvalue spectra may exhibit convergence instability. We demonstrate that activation function selection remains inherently problem-specific, with different bases showing distinct advantages for particular PDE characteristics. We believe these insights will help in the design of more robust neural PDE solvers.

Related papers

• Multi-level datasets training method in Physics-Informed Neural Networks [0.0]
  PINNs struggle with the challenging problems which are stiff to be solved and/or have high-frequency components in the solutions. In this study, an alternative approach is proposed to mitigate the above-mentioned problems. Inspired by the multi-grid method in CFD community, the underlying idea of the current approach is to efficiently remove different frequency errors via training.
  arXiv  Detail & Related papers  (2025-04-30T05:30:27Z)
• Towards a Foundation Model for Physics-Informed Neural Networks: Multi-PDE Learning with Active Sampling [0.0]
  Physics-Informed Neural Networks (PINNs) have emerged as a powerful framework for solving partial differential equations (PDEs) by embedding physical laws into neural network training. In this work, we explore the potential of a foundation PINN model capable of solving multiple PDEs within a unified architecture.
  arXiv  Detail & Related papers  (2025-02-11T10:12:28Z)
• Low Tensor-Rank Adaptation of Kolmogorov--Arnold Networks [70.06682043272377]
  Kolmogorov--Arnold networks (KANs) have demonstrated their potential as an alternative to multi-layer perceptions (MLPs) in various domains. We develop low tensor-rank adaptation (LoTRA) for fine-tuning KANs. We explore the application of LoTRA for efficiently solving various partial differential equations (PDEs) by fine-tuning KANs.
  arXiv  Detail & Related papers  (2025-02-10T04:57:07Z)
• AL-PINN: Active Learning-Driven Physics-Informed Neural Networks for Efficient Sample Selection in Solving Partial Differential Equations [0.0]
  Physics-Informed Neural Networks (PINNs) have emerged as a promising approach for solving Partial Differential Equations (PDEs) We propose Active Learning-Driven PINNs (AL-PINN), which integrates Uncertainty Quantification (UQ) and Active Learning strategies to optimize sample selection dynamically. Our results demonstrate that AL-PINN achieves comparable or superior accuracy compared to traditional PINNs while reducing the number of required training samples.
  arXiv  Detail & Related papers  (2025-02-06T10:54:28Z)
• SPIKANs: Separable Physics-Informed Kolmogorov-Arnold Networks [0.9999629695552196]
  Physics-Informed Neural Networks (PINNs) have emerged as a promising method for solving partial differential equations (PDEs) We introduce Separable Physics-Informed Kolmogorov-Arnold Networks (SPIKANs) This novel architecture applies the principle of separation of variables to PIKANs, decomposing the problem such that each dimension is handled by an individual KAN.
  arXiv  Detail & Related papers  (2024-11-09T21:10:23Z)
• DimOL: Dimensional Awareness as A New 'Dimension' in Operator Learning [63.5925701087252]
  We introduce DimOL (Dimension-aware Operator Learning), drawing insights from dimensional analysis. To implement DimOL, we propose the ProdLayer, which can be seamlessly integrated into FNO-based and Transformer-based PDE solvers. Empirically, DimOL models achieve up to 48% performance gain within the PDE datasets.
  arXiv  Detail & Related papers  (2024-10-08T10:48:50Z)
• DeltaPhi: Learning Physical Trajectory Residual for PDE Solving [54.13671100638092]
  We propose and formulate the Physical Trajectory Residual Learning (DeltaPhi) We learn the surrogate model for the residual operator mapping based on existing neural operator networks. We conclude that, compared to direct learning, physical residual learning is preferred for PDE solving.
  arXiv  Detail & Related papers  (2024-06-14T07:45:07Z)
• VS-PINN: A fast and efficient training of physics-informed neural networks using variable-scaling methods for solving PDEs with stiff behavior [0.0]
  We propose a new method for training PINNs using variable-scaling techniques. We will demonstrate the effectiveness of the proposed method for these problems and confirm that it can significantly improve the training efficiency and performance of PINNs.
  arXiv  Detail & Related papers  (2024-06-10T14:11:15Z)
• Enhancing Solutions for Complex PDEs: Introducing Complementary Convolution and Equivariant Attention in Fourier Neural Operators [17.91230192726962]
  We propose a novel hierarchical Fourier neural operator along with convolution-residual layers and attention mechanisms to solve complex PDEs. We find that the proposed method achieves superior performance in these PDE benchmarks, especially for equations characterized by rapid coefficient variations.
  arXiv  Detail & Related papers  (2023-11-21T11:04:13Z)
• Transport Equation based Physics Informed Neural Network to predict the Yield Strength of Architected Materials [0.0]
  The PINN model showcases exceptional generalization capabilities, indicating its capacity to avoid overfitting with the provided dataset. The research underscores the importance of striking a balance between performance and computational efficiency while selecting an activation function for specific real-world applications.
  arXiv  Detail & Related papers  (2023-07-29T12:42:03Z)
• Auxiliary-Tasks Learning for Physics-Informed Neural Network-Based Partial Differential Equations Solving [13.196871939441273]
  Physics-informed neural networks (PINNs) have emerged as promising surrogate modes for solving partial differential equations (PDEs) We propose auxiliary-task learning-based ATL-PINNs, which provide four different auxiliary-task learning modes. Our findings show that the proposed auxiliary-task learning modes can significantly improve solution accuracy, achieving a maximum performance boost of 96.62%.
  arXiv  Detail & Related papers  (2023-07-12T13:46:40Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Neural tangent kernel analysis of PINN for advection-diffusion equation [0.0]
  Physics-informed neural networks (PINNs) numerically approximate the solution of a partial differential equation (PDE) PINNs are known to struggle even in simple cases where the closed-form analytical solution is available. This work focuses on a systematic analysis of PINNs for the linear advection-diffusion equation (LAD) using the Neural Tangent Kernel (NTK) theory.
  arXiv  Detail & Related papers  (2022-11-21T18:35:14Z)
• Mitigating spectral bias for the multiscale operator learning [14.404769413313371]
  We propose a hierarchical attention neural operator (HANO) inspired by the hierarchical matrix approach. HANO features a scale-adaptive interaction range and self-attentions over a hierarchy of levels, enabling nested feature computation with controllable linear cost. Our numerical experiments demonstrate that HANO outperforms state-of-the-art (SOTA) methods for representative multiscale problems.
  arXiv  Detail & Related papers  (2022-10-19T21:09:29Z)
• Tunable Complexity Benchmarks for Evaluating Physics-Informed Neural Networks on Coupled Ordinary Differential Equations [64.78260098263489]
  In this work, we assess the ability of physics-informed neural networks (PINNs) to solve increasingly-complex coupled ordinary differential equations (ODEs) We show that PINNs eventually fail to produce correct solutions to these benchmarks as their complexity increases. We identify several reasons why this may be the case, including insufficient network capacity, poor conditioning of the ODEs, and high local curvature, as measured by the Laplacian of the PINN loss.
  arXiv  Detail & Related papers  (2022-10-14T15:01:32Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecture [77.59766598165551]
  Physics-informed neural networks (PINNs) are revolutionizing science and engineering practice by bringing together the power of deep learning to bear on scientific computation. Here, we propose Auto-PINN, which employs Neural Architecture Search (NAS) techniques to PINN design. A comprehensive set of pre-experiments using standard PDE benchmarks allows us to probe the structure-performance relationship in PINNs.
  arXiv  Detail & Related papers  (2022-05-27T03:24:31Z)
• Neural Operator with Regularity Structure for Modeling Dynamics Driven by SPDEs [70.51212431290611]
  Partial differential equations (SPDEs) are significant tools for modeling dynamics in many areas including atmospheric sciences and physics. We propose the Neural Operator with Regularity Structure (NORS) which incorporates the feature vectors for modeling dynamics driven by SPDEs. We conduct experiments on various of SPDEs including the dynamic Phi41 model and the 2d Navier-Stokes equation.
  arXiv  Detail & Related papers  (2022-04-13T08:53:41Z)
• Message Passing Neural PDE Solvers [60.77761603258397]
  We build a neural message passing solver, replacing allally designed components in the graph with backprop-optimized neural function approximators. We show that neural message passing solvers representationally contain some classical methods, such as finite differences, finite volumes, and WENO schemes. We validate our method on various fluid-like flow problems, demonstrating fast, stable, and accurate performance across different domain topologies, equation parameters, discretizations, etc., in 1D and 2D.
  arXiv  Detail & Related papers  (2022-02-07T17:47:46Z)
• Characterizing possible failure modes in physics-informed neural networks [55.83255669840384]
  Recent work in scientific machine learning has developed so-called physics-informed neural network (PINN) models. We demonstrate that, while existing PINN methodologies can learn good models for relatively trivial problems, they can easily fail to learn relevant physical phenomena even for simple PDEs. We show that these possible failure modes are not due to the lack of expressivity in the NN architecture, but that the PINN's setup makes the loss landscape very hard to optimize.
  arXiv  Detail & Related papers  (2021-09-02T16:06:45Z)
• A Physics Informed Neural Network Approach to Solution and Identification of Biharmonic Equations of Elasticity [0.0]
  We explore an application of the Physics Informed Neural Networks (PINNs) in conjunction with Airy stress functions and Fourier series. We find that enriching feature space using Airy stress functions can significantly improve the accuracy of PINN solutions for biharmonic PDEs.
  arXiv  Detail & Related papers  (2021-08-16T17:19:50Z)
• On the eigenvector bias of Fourier feature networks: From regression to solving multi-scale PDEs with physics-informed neural networks [0.0]
  We show that neural networks (PINNs) struggle in cases where the target functions to be approximated exhibit high-frequency or multi-scale features. We construct novel architectures that employ multi-scale random observational features and justify how such coordinate embedding layers can lead to robust and accurate PINN models.
  arXiv  Detail & Related papers  (2020-12-18T04:19:30Z)
• Multipole Graph Neural Operator for Parametric Partial Differential Equations [57.90284928158383]
  One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data. We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity. Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
  arXiv  Detail & Related papers  (2020-06-16T21:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.