ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks

• URL: http://arxiv.org/abs/2408.11104v1
• Date: Tue, 20 Aug 2024 18:00:20 GMT
• Title: ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks
• Authors: Qiang Liu, Mengyu Chu, Nils Thuerey,
• Abstract summary: We propose the ConFIG method to improve learning the physics-informed Neural Networks (PINNs) task. It provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels.
• Score: 25.333488397742432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. The proposed method is evaluated across a range of challenging PINN scenarios, consistently showing superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at \url{https://tum-pbs.github.io/ConFIG}.

Related papers

• Multi-level datasets training method in Physics-Informed Neural Networks [0.0]
  PINNs struggle with the challenging problems which are stiff to be solved and/or have high-frequency components in the solutions. In this study, an alternative approach is proposed to mitigate the above-mentioned problems. Inspired by the multi-grid method in CFD community, the underlying idea of the current approach is to efficiently remove different frequency errors via training.
  arXiv  Detail & Related papers  (2025-04-30T05:30:27Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture. Non-smooth regularization is often incorporated into machine learning tasks. We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Continual Optimization with Symmetry Teleportation for Multi-Task Learning [73.28772872740744]
  Multi-task learning (MTL) enables the simultaneous learning of multiple tasks using a single model. We propose a novel approach based on Continual Optimization with Symmetry Teleportation (COST) COST seeks an alternative loss-equivalent point on the loss landscape to reduce conflict gradients.
  arXiv  Detail & Related papers  (2025-03-06T02:58:09Z)
• Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective [12.712238596012742]
  We present theoretical and practical approaches for addressing directional conflicts between loss terms. We show how these conflicts limit first-order methods and show that second-order optimization naturally resolves them. We prove that SOAP, a recently proposed quasi-Newton method, efficiently approximates the Hessian preconditioner.
  arXiv  Detail & Related papers  (2025-02-02T00:21:45Z)
• Causal Context Adjustment Loss for Learned Image Compression [72.7300229848778]
  In recent years, learned image compression (LIC) technologies have surpassed conventional methods notably in terms of rate-distortion (RD) performance. Most present techniques are VAE-based with an autoregressive entropy model, which obviously promotes the RD performance by utilizing the decoded causal context. In this paper, we make the first attempt in investigating the way to explicitly adjust the causal context with our proposed Causal Context Adjustment loss.
  arXiv  Detail & Related papers  (2024-10-07T09:08:32Z)
• DiffGrad for Physics-Informed Neural Networks [0.0]
  Burgers' equation, a fundamental equation in fluid dynamics that is extensively used in PINNs, provides flexible results with the Adamprop. This paper introduces a novel strategy for solving Burgers' equation by incorporating DiffGrad with PINNs.
  arXiv  Detail & Related papers  (2024-09-05T04:39:35Z)
• Unveiling the optimization process of Physics Informed Neural Networks: How accurate and competitive can PINNs be? [0.0]
  This study investigates the potential accuracy of physics-informed neural networks, contrasting their approach with previous similar works and traditional numerical methods. We find that selecting improved optimization algorithms significantly enhances the accuracy of the results. Simple modifications to the loss function may also improve precision, offering an additional avenue for enhancement.
  arXiv  Detail & Related papers  (2024-05-07T11:50:25Z)
• Stabilizing Backpropagation Through Time to Learn Complex Physics [21.850601375335074]
  In physics simulations, backpropagating feedback is crucial to acquiring temporally coherent behavior. The alternative vector field we propose follows from two principles: physics simulators have a balanced gradient flow, and certain modifications to the backpropagation pass leave the positions of the original minima unchanged. Our final procedure is easily implementable via a sequence of gradient stopping and component-wise comparison operations.
  arXiv  Detail & Related papers  (2024-05-03T12:20:08Z)
• Optimizing Solution-Samplers for Combinatorial Problems: The Landscape of Policy-Gradient Methods [52.0617030129699]
  We introduce a novel theoretical framework for analyzing the effectiveness of DeepMatching Networks and Reinforcement Learning methods. Our main contribution holds for a broad class of problems including Max-and Min-Cut, Max-$k$-Bipartite-Bi, Maximum-Weight-Bipartite-Bi, and Traveling Salesman Problem. As a byproduct of our analysis we introduce a novel regularization process over vanilla descent and provide theoretical and experimental evidence that it helps address vanishing-gradient issues and escape bad stationary points.
  arXiv  Detail & Related papers  (2023-10-08T23:39:38Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Mixed formulation of physics-informed neural networks for thermo-mechanically coupled systems and heterogeneous domains [0.0]
  Physics-informed neural networks (PINNs) are a new tool for solving boundary value problems. Recent investigations have shown that when designing loss functions for many engineering problems, using first-order derivatives and combining equations from both strong and weak forms can lead to much better accuracy. In this work, we propose applying the mixed formulation to solve multi-physical problems, specifically a stationary thermo-mechanically coupled system of equations.
  arXiv  Detail & Related papers  (2023-02-09T21:56:59Z)
• Adaptive Self-supervision Algorithms for Physics-informed Neural Networks [59.822151945132525]
  Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function. We study the impact of the location of the collocation points on the trainability of these models. We propose a novel adaptive collocation scheme which progressively allocates more collocation points to areas where the model is making higher errors.
  arXiv  Detail & Related papers  (2022-07-08T18:17:06Z)
• Joint inference and input optimization in equilibrium networks [68.63726855991052]
  deep equilibrium model is a class of models that foregoes traditional network depth and instead computes the output of a network by finding the fixed point of a single nonlinear layer. We show that there is a natural synergy between these two settings. We demonstrate this strategy on various tasks such as training generative models while optimizing over latent codes, training models for inverse problems like denoising and inpainting, adversarial training and gradient based meta-learning.
  arXiv  Detail & Related papers  (2021-11-25T19:59:33Z)
• Multi-Objective Loss Balancing for Physics-Informed Deep Learning [0.0]
  We observe the role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. We propose a novel self-adaptive loss balancing of PINNs called ReLoBRaLo. Our simulation studies show that ReLoBRaLo training is much faster and achieves higher accuracy than training PINNs with other balancing methods.
  arXiv  Detail & Related papers  (2021-10-19T09:00:12Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.