Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks

• URL: http://arxiv.org/abs/2606.20442v1
• Date: Thu, 18 Jun 2026 16:22:52 GMT
• Title: Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks
• Abstract summary: We propose to investigate a two-stage approach based on evolutionary algorithms that combines parts of training and solution accuracy.<n>We evaluate on three popular problems, and our solution outperforms standard training and achieves within computational resources.
• Score: 0.6924349411126935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physics-Informed Neural Networks (PINNs) solve Partial Differential Equations (PDEs) by embedding physical laws into neural network training. However, their performance suffers from unstable convergence, training plateaus, and strong sensitivity to architectural and optimization hyperparameters due to the highly non-convex and multi-term structure of the physics-informed loss. In this setting, the outer-loop hyperparameter search is a noisy and black-box optimization problem over heterogeneous parameters, where classical local or gradient-based strategies are easily trapped in suboptimal regions. Evolutionary algorithms, with their population-based exploration and ability to handle mixed, non-differentiable search spaces, provide a more robust mechanism for discovering promising configurations. We propose and investigate a two-stage approach based on evolutionary algorithms that combines exploration and exploitation parts of PINNs training to improve solution accuracy and robustness under fixed computational budgets. In the first stage, we perform low-fidelity training runs with truncated epochs to rapidly screen candidate configurations, treating hyperparameter selection as a black-box outer-loop problem. In the second stage, only the most promising candidates are fully trained with standard gradient-based optimizers to refine the solution. Evaluated on three popular problems, namely Advection, Klein-Gordon and Helmholtz equations, our method consistently outperforms standard training and achieves significantly lower mean error within constrained computational resources.

Related papers

• Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations [98.44542103979735]
  First-order gradient descent methods, epitomized by gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines.<n>Large-scale model training, stringent privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency.<n>We retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstreams across diverse model architectures and training scenarios.<n>We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research.
  arXiv  Detail & Related papers  (2026-04-14T17:01:36Z)
• Curvature-Aware Optimization for High-Accuracy Physics-Informed Neural Networks [6.7049043613619155]
  We present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs)<n>We demonstrate their performance on problems including Helmholtz equation, Stokes flow, inviscid Burgers equation, Euler equations for high-speed flows.<n>We also propose new PINN-based methods for solving the inviscid Burgers equation, and compare the resulting solutions against high-order numerical methods.
  arXiv  Detail & Related papers  (2026-04-06T22:52:34Z)
• Neural network optimization strategies and the topography of the loss landscape [45.88028371034407]
  We investigate neural network learning by gradient descent (SGD)<n>We use several computational tools to investigate neural network parameters obtained by these two optimization methods.
  arXiv  Detail & Related papers  (2026-02-24T17:49:13Z)
• Implicitly Restarted Lanczos Enables Chemically-Accurate Shallow Neural Quantum States [4.039934762896615]
  We introduce the implicitly restarted Lanczos (IRL) method as the core engine for neural quantum states (NQS) training.<n>Our key innovation is an inherently stable second-order optimization framework that recasts the ill-conditioned parameter update problem into a small, well-posed Hermitian eigenvalue problem.<n>We demonstrate that IRL enables shallow NQS to consistently achieve extreme precision (1e-12 kcal/mol) in just 3 to 5 optimization steps.<n>For the F2 molecule, this translates to an approximate 17,900-fold speed-up in total runtime compared to Adam.
  arXiv  Detail & Related papers  (2026-01-04T08:49:24Z)
• MAD-NG: Meta-Auto-Decoder Neural Galerkin Method for Solving Parametric Partial Differential Equations [5.767740428776141]
  Parametric partial differential equations (PDEs) are fundamental for modeling a wide range of physical and engineering systems.<n>Traditional neural network-based solvers, such as Physics-Informed Neural Networks (PINNs) and Deep Galerkin Methods, often face challenges in generalization and long-time prediction efficiency.<n>We propose a novel and scalable framework that significantly enhances the Neural Galerkin Method (NGM) by incorporating the Meta-Auto-Decoder (MAD) paradigm.
  arXiv  Detail & Related papers  (2025-12-25T11:27:40Z)
• Optimizing Optimizers for Fast Gradient-Based Learning [53.81268610971847]
  We lay the theoretical foundation for automating design in gradient-based learning.<n>By treating gradient loss signals as a function that translates to parameter motions, the problem reduces to a family of convex optimization problems.
  arXiv  Detail & Related papers  (2025-12-06T09:50:41Z)
• SO-PIFRNN: Self-optimization physics-informed Fourier-features randomized neural network for solving partial differential equations [3.769992289689535]
  This study proposes a self-optimization physics-informed Fourier-features randomized neural network (SO-PIFRNN) framework.<n>The inner-level optimization determines the output layer weights of the neural network via the least squares method.<n>The experimental results affirm that SO-PIFRNN exhibits superior approximation accuracy and frequency capture capability.
  arXiv  Detail & Related papers  (2025-08-07T02:08:34Z)
• Optimizing the Optimizer for Physics-Informed Neural Networks and Kolmogorov-Arnold Networks [3.758814046658822]
  Physics-Informed Neural Networks (PINNs) have revolutionized the computation PDE solutions by integrating partialmagnitude equations (PDEs) into the neural network's training process as soft constraints.<n>More, physics-informed networks (PIKANs) have also been effective and comparable in accuracy.
  arXiv  Detail & Related papers  (2025-01-22T21:19:42Z)
• Enhancing CNN Classification with Lamarckian Memetic Algorithms and Local Search [0.0]
  We propose a novel approach integrating a two-stage training technique with population-based optimization algorithms incorporating local search capabilities. Our experiments demonstrate that the proposed method outperforms state-of-the-art gradient-based techniques.
  arXiv  Detail & Related papers  (2024-10-26T17:31:15Z)
• A Stochastic Approach to Bi-Level Optimization for Hyperparameter Optimization and Meta Learning [74.80956524812714]
  We tackle the general differentiable meta learning problem that is ubiquitous in modern deep learning. These problems are often formalized as Bi-Level optimizations (BLO) We introduce a novel perspective by turning a given BLO problem into a ii optimization, where the inner loss function becomes a smooth distribution, and the outer loss becomes an expected loss over the inner distribution.
  arXiv  Detail & Related papers  (2024-10-14T12:10:06Z)
• A Deep Unrolling Model with Hybrid Optimization Structure for Hyperspectral Image Deconvolution [50.13564338607482]
  We propose a novel optimization framework for the hyperspectral deconvolution problem, called DeepMix.<n>It consists of three distinct modules, namely, a data consistency module, a module that enforces the effect of the handcrafted regularizers, and a denoising module.<n>This work proposes a context aware denoising module designed to sustain the advancements achieved by the cooperative efforts of the other modules.
  arXiv  Detail & Related papers  (2023-06-10T08:25:16Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• NeuralStagger: Accelerating Physics-constrained Neural PDE Solver with Spatial-temporal Decomposition [67.46012350241969]
  This paper proposes a general acceleration methodology called NeuralStagger. It decomposing the original learning tasks into several coarser-resolution subtasks. We demonstrate the successful application of NeuralStagger on 2D and 3D fluid dynamics simulations.
  arXiv  Detail & Related papers  (2023-02-20T19:36:52Z)
• Neuroevolution of Physics-Informed Neural Nets: Benchmark Problems and Comparative Results [25.12291688711645]
  Physics-informed neural networks (PINNs) are one of the key techniques at the forefront of recent advances. PINNs' unique loss formulations lead to a high degree of complexity and ruggedness that may not be conducive for gradient descent. Neuroevolution algorithms, with their superior global search capacity, may be a better choice for PINNs.
  arXiv  Detail & Related papers  (2022-12-15T05:54:16Z)
• Message Passing Neural PDE Solvers [60.77761603258397]
  We build a neural message passing solver, replacing allally designed components in the graph with backprop-optimized neural function approximators. We show that neural message passing solvers representationally contain some classical methods, such as finite differences, finite volumes, and WENO schemes. We validate our method on various fluid-like flow problems, demonstrating fast, stable, and accurate performance across different domain topologies, equation parameters, discretizations, etc., in 1D and 2D.
  arXiv  Detail & Related papers  (2022-02-07T17:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.