Mask-PINNs: Mitigating Internal Covariate Shift in Physics-Informed Neural Networks

• URL: http://arxiv.org/abs/2505.06331v3
• Date: Mon, 01 Sep 2025 14:38:13 GMT
• Title: Mask-PINNs: Mitigating Internal Covariate Shift in Physics-Informed Neural Networks
• Authors: Feilong Jiang, Xiaonan Hou, Jianqiao Ye, Min Xia,
• Abstract summary: PINNs have emerged as a powerful framework for solving partial differential equations.<n>We propose Mask-PINNs, a learnable mask function to regulate feature distributions.<n>Our results show consistent improvements in prediction accuracy, convergence stability, and robustness.
• Score: 1.2667864219315372
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physics-Informed Neural Networks (PINNs) have emerged as a powerful framework for solving partial differential equations (PDEs) by embedding physical laws directly into the loss function. However, as a fundamental optimization issue, internal covariate shift (ICS) hinders the stable and effective training of PINNs by disrupting feature distributions and limiting model expressiveness. Unlike standard deep learning tasks, conventional remedies for ICS -- such as Batch Normalization and Layer Normalization -- are not directly applicable to PINNs, as they distort the physical consistency required for reliable PDE solutions. To address this issue, we propose Mask-PINNs, a novel architecture that introduces a learnable mask function to regulate feature distributions while preserving the underlying physical constraints of PINNs. We provide a theoretical analysis showing that the mask suppresses the expansion of feature representations through a carefully designed modulation mechanism. Empirically, we validate the method on multiple PDE benchmarks -- including convection, wave propagation, and Helmholtz equations -- across diverse activation functions. Our results show consistent improvements in prediction accuracy, convergence stability, and robustness. Furthermore, we demonstrate that Mask-PINNs enable the effective use of wider networks, overcoming a key limitation in existing PINN frameworks.

Related papers

• Forked Physics Informed Neural Networks for Coupled Systems of Differential equations [1.7158443622461192]
  We propose a Forked PINN framework designed for coupled systems of differential equations (DEs)<n>We demonstrate the effectiveness of FPINN in simulating non-Markovian open quantum dynamics governed by coupled DEs.<n>For the spin-boson and XXZ models, FPINN accurately captures non-Markovian features, such as quantum coherence revival and information backflow.
  arXiv  Detail & Related papers  (2026-02-16T08:32:59Z)
• LNN-PINN: A Unified Physics-Only Training Framework with Liquid Residual Blocks [1.6249267147413524]
  LNN-PINN is a physics-informed neural network framework that incorporates a liquid residual gating architecture.<n>Across four benchmark problems, LNN-PINN consistently reduced RMSE and MAE under identical training conditions.
  arXiv  Detail & Related papers  (2025-08-12T13:35:46Z)
• KITINet: Kinetics Theory Inspired Network Architectures with PDE Simulation Approaches [43.872190335490515]
  This paper introduces KITINet, a novel architecture that reinterprets feature propagation through the lens of non-equilibrium particle dynamics.<n>At its core, we propose a residual module that models update as the evolution of a particle system.<n>This formulation mimics particle collisions and energy exchange, enabling adaptive feature refinement via physics-informed interactions.
  arXiv  Detail & Related papers  (2025-05-23T13:58:29Z)
• ProPINN: Demystifying Propagation Failures in Physics-Informed Neural Networks [71.02216400133858]
  Physics-informed neural networks (PINNs) have earned high expectations in solving partial differential equations (PDEs)<n>Previous research observed the propagation failure phenomenon of PINNs.<n>This paper provides a formal and in-depth study of propagation failure and its root cause.
  arXiv  Detail & Related papers  (2025-02-02T13:56:38Z)
• Finite Element Neural Network Interpolation. Part I: Interpretable and Adaptive Discretization for Solving PDEs [44.99833362998488]
  We present a sparse neural network architecture extending previous work on Embedded Finite Element Neural Networks (EFENN)<n>Due to their mesh-based structure, EFENN requires significantly fewer trainable parameters than fully connected neural networks.<n>Our FENNI framework, within the EFENN framework, brings improvements to the HiDeNN approach.
  arXiv  Detail & Related papers  (2024-12-07T18:31:17Z)
• HyResPINNs: Hybrid Residual Networks for Adaptive Neural and RBF Integration in Solving PDEs [22.689531776611084]
  We introduce HyResPINNs, a novel class of PINNs featuring adaptive hybrid residual blocks that integrate standard neural networks and radial basis function networks.<n>A distinguishing characteristic of HyResPINNs is the use of adaptive combination parameters within each residual block, enabling dynamic weighting of the neural and RBF network contributions.
  arXiv  Detail & Related papers  (2024-10-04T16:21:14Z)
• PseudoNeg-MAE: Self-Supervised Point Cloud Learning using Conditional Pseudo-Negative Embeddings [55.55445978692678]
  PseudoNeg-MAE enhances global feature representation of point cloud masked autoencoders by making them both discriminative and sensitive to transformations.<n>We propose a novel loss that explicitly penalizes invariant collapse, enabling the network to capture richer transformation cues while preserving discriminative representations.
  arXiv  Detail & Related papers  (2024-09-24T07:57:21Z)
• ASPINN: An asymptotic strategy for solving singularly perturbed differential equations [12.14934707131722]
  We propose Asymptotic Physics-Informed Neural Networks (ASPINN), a generalization of Physics-Informed Neural Networks (PINN) and General-Kindred Physics-Informed Neural Networks (GKPINN) ASPINN has a strong fitting ability for solving SPDEs due to the placement of exponential layers at the boundary layer. We demonstrate the effect of ASPINN by solving diverse classes of SPDEs, which clearly shows that the ASPINN method is promising in boundary layer problems.
  arXiv  Detail & Related papers  (2024-09-20T03:25:17Z)
• Physics-Informed Neural Networks with Trust-Region Sequential Quadratic Programming [4.557963624437784]
  Recent research has noted that Physics-Informed Neural Networks (PINNs) may fail to learn relatively complex Partial Differential Equations (PDEs) This paper addresses the failure modes of PINNs by introducing a novel, hard-constrained deep learning method -- trust-region Sequential Quadratic Programming (trSQP-PINN) In contrast to directly training the penalized soft-constrained loss as in PINNs, our method performs a linear-quadratic approximation of the hard-constrained loss, while leveraging the soft-constrained loss to adaptively adjust the trust-region radius.
  arXiv  Detail & Related papers  (2024-09-16T23:22:12Z)
• Residual resampling-based physics-informed neural network for neutron diffusion equations [7.105073499157097]
  The neutron diffusion equation plays a pivotal role in the analysis of nuclear reactors. Traditional PINN approaches often utilize fully connected network (FCN) architecture. R2-PINN effectively overcomes the limitations inherent in current methods, providing more accurate and robust solutions for neutron diffusion equations.
  arXiv  Detail & Related papers  (2024-06-23T13:49:31Z)
• Collocation-based Robust Variational Physics-Informed Neural Networks (CRVPINN) [0.0]
  Physics-Informed Neural Networks (PINNs) have been successfully applied to solve Partial Differential Equations (PDEs) Recent work of Robust Variational Physics-Informed Neural Networks (RVPINNs) highlights the importance of conveniently translating the norms of the underlying continuum-level spaces to the discrete level. In this work, we accelerate the implementation of RVPINN, establishing a LU factorization of sparse Gram matrix in a kind of point-collocation scheme with the same spirit as original PINNs.
  arXiv  Detail & Related papers  (2024-01-04T14:42:29Z)
• Achieving Constraints in Neural Networks: A Stochastic Augmented Lagrangian Approach [49.1574468325115]
  Regularizing Deep Neural Networks (DNNs) is essential for improving generalizability and preventing overfitting. We propose a novel approach to DNN regularization by framing the training process as a constrained optimization problem. We employ the Augmented Lagrangian (SAL) method to achieve a more flexible and efficient regularization mechanism.
  arXiv  Detail & Related papers  (2023-10-25T13:55:35Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  We present Layer-wise Feedback Propagation (LFP), a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions to solving a given task.<n>Our method then implements a greedy approach reinforcing helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• PINNsFormer: A Transformer-Based Framework For Physics-Informed Neural Networks [22.39904196850583]
  Physics-Informed Neural Networks (PINNs) have emerged as a promising deep learning framework for approximating numerical solutions to partial differential equations (PDEs) We introduce a novel Transformer-based framework, termed PINNsFormer, designed to address this limitation. PINNsFormer achieves superior generalization ability and accuracy across various scenarios, including PINNs failure modes and high-dimensional PDEs.
  arXiv  Detail & Related papers  (2023-07-21T18:06:27Z)
• Input-gradient space particle inference for neural network ensembles [32.64178604645513]
  First-order Repulsive Deep Ensemble (FoRDE) is an ensemble learning method based on ParVI. Experiments on image classification datasets and transfer learning tasks show that FoRDE significantly outperforms the gold-standard DEs.
  arXiv  Detail & Related papers  (2023-06-05T11:00:11Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Permutation Equivariant Neural Functionals [92.0667671999604]
  This work studies the design of neural networks that can process the weights or gradients of other neural networks. We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order. In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-02-27T18:52:38Z)
• Ensemble learning for Physics Informed Neural Networks: a Gradient Boosting approach [10.250994619846416]
  We present a new training paradigm referred to as "gradient boosting" (GB) Instead of learning the solution of a given PDE using a single neural network directly, our algorithm employs a sequence of neural networks to achieve a superior outcome. This work also unlocks the door to employing ensemble learning techniques in PINNs.
  arXiv  Detail & Related papers  (2023-02-25T19:11:44Z)
• Auto-PINN: Understanding and Optimizing Physics-Informed Neural Architecture [77.59766598165551]
  Physics-informed neural networks (PINNs) are revolutionizing science and engineering practice by bringing together the power of deep learning to bear on scientific computation. Here, we propose Auto-PINN, which employs Neural Architecture Search (NAS) techniques to PINN design. A comprehensive set of pre-experiments using standard PDE benchmarks allows us to probe the structure-performance relationship in PINNs.
  arXiv  Detail & Related papers  (2022-05-27T03:24:31Z)
• Revisiting PINNs: Generative Adversarial Physics-informed Neural Networks and Point-weighting Method [70.19159220248805]
  Physics-informed neural networks (PINNs) provide a deep learning framework for numerically solving partial differential equations (PDEs) We propose the generative adversarial neural network (GA-PINN), which integrates the generative adversarial (GA) mechanism with the structure of PINNs. Inspired from the weighting strategy of the Adaboost method, we then introduce a point-weighting (PW) method to improve the training efficiency of PINNs.
  arXiv  Detail & Related papers  (2022-05-18T06:50:44Z)
• Learning Physics-Informed Neural Networks without Stacked Back-propagation [82.26566759276105]
  We develop a novel approach that can significantly accelerate the training of Physics-Informed Neural Networks. In particular, we parameterize the PDE solution by the Gaussian smoothed model and show that, derived from Stein's Identity, the second-order derivatives can be efficiently calculated without back-propagation. Experimental results show that our proposed method can achieve competitive error compared to standard PINN training but is two orders of magnitude faster.
  arXiv  Detail & Related papers  (2022-02-18T18:07:54Z)
• Robust Learning of Physics Informed Neural Networks [2.86989372262348]
  Physics-informed Neural Networks (PINNs) have been shown to be effective in solving partial differential equations. This paper shows that a PINN can be sensitive to errors in training data and overfit itself in dynamically propagating these errors over the domain of the solution of the PDE.
  arXiv  Detail & Related papers  (2021-10-26T00:10:57Z)
• Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism [1.6114012813668932]
  Physics-Informed Neural Networks (PINNs) have emerged as a promising application of deep neural networks to the numerical solution of nonlinear partial differential equations (PDEs) We propose a fundamentally new way to train PINNs adaptively, where the adaptation weights are fully trainable and applied to each training point individually. In numerical experiments with several linear and nonlinear benchmark problems, the SA-PINN outperformed other state-of-the-art PINN algorithm in L2 error.
  arXiv  Detail & Related papers  (2020-09-07T04:07:52Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• Learning Likelihoods with Conditional Normalizing Flows [54.60456010771409]
  Conditional normalizing flows (CNFs) are efficient in sampling and inference. We present a study of CNFs where the base density to output space mapping is conditioned on an input x, to model conditional densities p(y|x)
  arXiv  Detail & Related papers  (2019-11-29T19:17:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.