Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation
- URL: http://arxiv.org/abs/2507.16008v1
- Date: Mon, 21 Jul 2025 18:59:26 GMT
- Title: Enhancing Stability of Physics-Informed Neural Network Training Through Saddle-Point Reformulation
- Authors: Dmitry Bylinkin, Mikhail Aleksandrov, Savelii Chezhegov, Aleksandr Beznosikov,
- Abstract summary: Physics-informed neural networks (PINs) have gained prominence in recent years and are now effectively used in a number of applications.<n>To address this issue, we reformulate their landscape as a non-strongly concave-point problem.<n>Our results demonstrate that the proposed-of-the-art technique outperforms the current state-the-art techniques.
- Score: 44.31966204357333
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Physics-informed neural networks (PINNs) have gained prominence in recent years and are now effectively used in a number of applications. However, their performance remains unstable due to the complex landscape of the loss function. To address this issue, we reformulate PINN training as a nonconvex-strongly concave saddle-point problem. After establishing the theoretical foundation for this approach, we conduct an extensive experimental study, evaluating its effectiveness across various tasks and architectures. Our results demonstrate that the proposed method outperforms the current state-of-the-art techniques.
Related papers
- Stable Weight Updating: A Key to Reliable PDE Solutions Using Deep Learning [0.0]
This paper introduces novel residual-based architectures, designed to enhance stability and accuracy in physics-informed neural networks (PINNs)
The architectures augment traditional neural networks by incorporating residual connections, which facilitate smoother weight updates and improve backpropagation efficiency.
The Squared Residual Network, in particular, exhibits robust performance, achieving enhanced stability and accuracy compared to conventional neural networks.
arXiv Detail & Related papers (2024-07-10T05:20:43Z) - Architectural Strategies for the optimization of Physics-Informed Neural
Networks [30.92757082348805]
Physics-informed neural networks (PINNs) offer a promising avenue for tackling both forward and inverse problems in partial differential equations (PDEs)
Despite their remarkable empirical success, PINNs have garnered a reputation for their notorious training challenges across a spectrum of PDEs.
arXiv Detail & Related papers (2024-02-05T04:15:31Z) - Implicit Stochastic Gradient Descent for Training Physics-informed
Neural Networks [51.92362217307946]
Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems.
PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features.
In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
arXiv Detail & Related papers (2023-03-03T08:17:47Z) - Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach [10.250994619846416]
We present a new training paradigm referred to as "gradient boosting" (GB)
Instead of learning the solution of a given PDE using a single neural network directly, our algorithm employs a sequence of neural networks to achieve a superior outcome.
This work also unlocks the door to employing ensemble learning techniques in PINNs.
arXiv Detail & Related papers (2023-02-25T19:11:44Z) - How to Avoid Trivial Solutions in Physics-Informed Neural Networks [0.0]
We investigate the prediction performance of PINNs with respect to the number of collocation points used to enforce the physics-based penalty terms.
We show that PINNs can fail, learning a trivial solution that fulfills the physics-derived penalty term by definition.
We have developed an alternative sampling approach and a new penalty term enabling us to remedy this core problem of PINNs in data-scarce settings with competitive results.
arXiv Detail & Related papers (2021-12-10T15:54:54Z) - Physics-guided Loss Functions Improve Deep Learning Performance in
Inverse Scattering [13.529767949868248]
Deep neural network (DNN) techniques have been successfully applied on electromagnetic inverse scattering problems.
We show how important physical phenomena cannot be effectively incorporated in the training process.
We propose new designs of loss functions which incorporate multiple-scattering based near-field quantities.
arXiv Detail & Related papers (2021-11-13T16:36:23Z) - Physics informed neural networks for continuum micromechanics [68.8204255655161]
Recently, physics informed neural networks have successfully been applied to a broad variety of problems in applied mathematics and engineering.
Due to the global approximation, physics informed neural networks have difficulties in displaying localized effects and strong non-linear solutions by optimization.
It is shown, that the domain decomposition approach is able to accurately resolve nonlinear stress, displacement and energy fields in heterogeneous microstructures obtained from real-world $mu$CT-scans.
arXiv Detail & Related papers (2021-10-14T14:05:19Z) - A Theoretical Framework for Target Propagation [75.52598682467817]
We analyze target propagation (TP), a popular but not yet fully understood alternative to backpropagation (BP)
Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP.
We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training.
arXiv Detail & Related papers (2020-06-25T12:07:06Z) - Disturbance-immune Weight Sharing for Neural Architecture Search [96.93812980299428]
We propose a disturbance-immune update strategy for model updating.
We theoretically analyze the effectiveness of our strategy in alleviating the performance disturbance risk.
arXiv Detail & Related papers (2020-03-29T17:54:49Z) - Robust Pruning at Initialization [61.30574156442608]
A growing need for smaller, energy-efficient, neural networks to be able to use machine learning applications on devices with limited computational resources.
For Deep NNs, such procedures remain unsatisfactory as the resulting pruned networks can be difficult to train and, for instance, they do not prevent one layer from being fully pruned.
arXiv Detail & Related papers (2020-02-19T17:09:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.