Gradient Scaling Effects in Adaptive Spectral PINNs for Stiff Nonlinear ODEs

• URL: http://arxiv.org/abs/2605.04502v1
• Date: Wed, 06 May 2026 05:10:20 GMT
• Title: Gradient Scaling Effects in Adaptive Spectral PINNs for Stiff Nonlinear ODEs
• Abstract summary: We show that the choice of IC gating function induces explicit time-dependent gradient scaling.<n>We observe stiffness-dependent changes in relative dominance for adaptive PINNs at moderate stiffness.<n>These trends hold for both relative $L2$ error and pointwise error.
• Score: 0.794957965474334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physics-Informed Neural Networks (PINNs) often struggle to train reliably on stiff and oscillatory dynamical systems due to poor optimization conditioning. While prior work has emphasized representational remedies such as spectral parameterizations, the optimization implications of initial-condition (IC) embeddings in adaptive spectral PINNs have not been well characterized. In this work, we show that the choice of IC gating function induces explicit time-dependent gradient scaling, which interacts with spectral representations during training. Using a nonlinear stiff spring-pendulum ODE as a controlled benchmark, we compare exponential and linear IC gates in combination with fixed and adaptive Fourier spectral trunks. We observe stiffness-dependent changes in relative dominance for adaptive PINNs: at moderate stiffness ($k=20$), exponential gating often yields lower error but exhibits heterogeneous behavior across random seeds, whereas at higher stiffness ($k=60$), linear gating becomes preferable, with additional reversals observed at larger $k$. These trends hold for both relative $L^2$ error and maximum pointwise error and are confirmed by paired Wilcoxon signed-rank tests with Holm correction. Overall, our results demonstrate that IC embeddings are not a neutral design choice in PINNs: the induced gradient scaling materially shapes optimization conditioning in stiff regimes, with distinct sensitivity patterns in baseline and adaptive spectral models.

Related papers

• Variational Boosting for Physics-Informed Neural Networks [0.794957965474334]
  monolithic PINNs often suffer from ill-conditioning, spectral bias, and optimization instability.<n>We introduce a variational boosting framework in which solutions are constructed additively in function space.<n>This framework provides a geometric interpretation of multi-stage PINNs as projected functional gradient descent.
  arXiv  Detail & Related papers  (2026-07-27T02:32:22Z)
• From Simple to Complex: Curriculum-Guided Physics-Informed Neural Networks via Gaussian Mixture Models [13.535408662482434]
  Physics-informed neural networks (PINNs) offer a mesh-free framework for solving partial differential equations (PDEs)<n>We propose the CurriculumGuided Mixture Physics-Informed Neural Network (CGMPINN), which integrates mixture modeling with dynamic curriculum learning.
  arXiv  Detail & Related papers  (2026-05-19T02:20:13Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Structure-Preserving Physics-Informed Neural Network for the Korteweg--de Vries (KdV) Equation [0.0]
  This paper introduces a emphstructure-preserving PINN framework for the nonlinear Korteweg--de Vries (KdV) equation.<n>The proposed method embeds the conservation of mass and Hamiltonian energy directly into the loss function.<n>We successfully reproduces hallmark behaviors of KdV dynamics while maintaining conserved invariants.
  arXiv  Detail & Related papers  (2025-11-01T06:07:24Z)
• Curvature Learning for Generalization of Hyperbolic Neural Networks [51.888534247573894]
  Hyperbolic neural networks (HNNs) have demonstrated notable efficacy in representing real-world data with hierarchical structures.<n>Inappropriate curvatures may cause HNNs to converge to suboptimal parameters, degrading overall performance.<n>We propose a sharpness-aware curvature learning method to smooth the loss landscape, thereby improving the generalization of HNNs.
  arXiv  Detail & Related papers  (2025-08-24T07:14:30Z)
• PIG: Physics-Informed Gaussians as Adaptive Parametric Mesh Representations [5.4087282763977855]
  We propose Physics-Informed Gaussians (PIGs), which combine feature embeddings using Gaussian functions with a lightweight neural network.<n>Our approach uses trainable parameters for the mean and variance of each Gaussian, allowing for dynamic adjustment of their positions and shapes during training.<n> Experimental results show the competitive performance of our model across various PDEs, demonstrating its potential as a robust tool for solving complex PDEs.
  arXiv  Detail & Related papers  (2024-12-08T16:58:29Z)
• Gradient Normalization Provably Benefits Nonconvex SGD under Heavy-Tailed Noise [60.92029979853314]
  We investigate the roles of gradient normalization and clipping in ensuring the convergence of Gradient Descent (SGD) under heavy-tailed noise. Our work provides the first theoretical evidence demonstrating the benefits of gradient normalization in SGD under heavy-tailed noise. We introduce an accelerated SGD variant incorporating gradient normalization and clipping, further enhancing convergence rates under heavy-tailed noise.
  arXiv  Detail & Related papers  (2024-10-21T22:40:42Z)
• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures [14.93489065234423]
  We present a general variational framework for the training of freeform nonlinearities in layered computational architectures.<n>The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility.<n>We show how to solve the numerically function-optimization problem by representing the nonlinearities in a suitable (nonuniform) B-spline basis.
  arXiv  Detail & Related papers  (2024-08-23T14:39:27Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Momentum Diminishes the Effect of Spectral Bias in Physics-Informed Neural Networks [72.09574528342732]
  Physics-informed neural network (PINN) algorithms have shown promising results in solving a wide range of problems involving partial differential equations (PDEs) They often fail to converge to desirable solutions when the target function contains high-frequency features, due to a phenomenon known as spectral bias. In the present work, we exploit neural tangent kernels (NTKs) to investigate the training dynamics of PINNs evolving under gradient descent with momentum (SGDM)
  arXiv  Detail & Related papers  (2022-06-29T19:03:10Z)
• Optimization-Induced Graph Implicit Nonlinear Diffusion [64.39772634635273]
  We propose a new kind of graph convolution variants, called Graph Implicit Diffusion (GIND) GIND implicitly has access to infinite hops of neighbors while adaptively aggregating features with nonlinear diffusion to prevent over-smoothing. We show that the learned representation can be formalized as the minimizer of an explicit convex optimization objective.
  arXiv  Detail & Related papers  (2022-06-29T06:26:42Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.