SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning

• URL: http://arxiv.org/abs/2608.29448v1
• Date: Sat, 29 Aug 2026 21:34:47 GMT
• Title: SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning
• Abstract summary: method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
• Score: 70.71795707610961
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Physics-informed neural networks (PINNs) often face ill-conditioned objectives that limit high-accuracy training. Dense quasi-Newton methods improve local conditioning but require expensive optimizer state, while Kronecker-factored methods such as SOAP scale to larger networks but rely on periodic basis updates. We introduce \method, which augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry and an adaptive basis update followed by variance-state downscaling. We characterize the directional secant matching induced by the scalar correction and give a bound on variance-state mismatch across basis changes. Across eight PDE benchmarks, \method attains the lowest final residual on six, including Burgers and Boussinesq, while SOAP-family baselines perform better on Gray-Scott and Ginzburg-Landau. On Boussinesq, \method reaches a residual of $10^{-5}$ in 4.1 hours with 9.2 GB peak VRAM, while Adam does not reach this target within 14 hours. Three-seed $L^2$ and $H^1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy. These results position \method as a scalable option for stiff, high-accuracy physics-informed training, rather than a uniform replacement for existing optimizers.

Related papers

• Error-Conditioned Neural Solvers [36.51479208528461]
  Neural surrogate models offer fast approximate mappings from PDE parameters to solutions, but they typically treat solving as a purely statistical task.<n>Recent hybrid methods promote physical correctness by targeting the PDE residual via gradient descent or Gauss-Newton steps, but inherit the compute cost and instability of the underlying classical gradients.<n>We show, theoretically and empirically, that numerically minimizing the PDE residual can be an unreliable proxy for reconstruction accuracy in ill-conditioned systems.
  arXiv  Detail & Related papers  (2026-06-25T17:56:27Z)
• Singularity-aware Optimization via Randomized Geometric Probing: Towards Stable Non-smooth Optimization [8.412194372218895]
  We introduce Singularity-aware Adam (S-Adam), a novel that stabilizes training by dynamically modulating step sizes based on local geometric instability.<n>S-Adam incorporates an adaptive damping mechanism that decelerates updates in high-instability regions while preserving fast convergence in smooth basins.<n> Empirical evaluations on optimal Quantization-Aware Training (QAT) and high-noise small-batch learning demonstrate that S-Adam consistently outperforms AdamW and Prox-SGD.
  arXiv  Detail & Related papers  (2026-05-28T08:00:40Z)
• Not How Many, But Which: Parameter Placement in Low-Rank Adaptation [7.686602368047758]
  Given a fixed budget of $k$ trainable entries within the B matrix of a LoRA adapter (A frozen), does the choice of which $k$ matter?<n>Under supervised fine-tuning, random and informed subsets achieve comparable performance.<n>Under GRPO on base models, random placement fails to improve over the base model, while gradient-informed placement recovers standard LoRA accuracy.
  arXiv  Detail & Related papers  (2026-05-12T14:46:00Z)
• Beta-Scheduling: Momentum from Critical Damping as a Diagnostic and Correction Tool for Neural Network Training [0.0]
  Standard neural network training uses constant momentum (typically 0.9), a convention dating to 1964 with limited theoretical justification for its optimality.<n>This beta-schedule requires zero free parameters beyond the existing learning rate schedule.<n>On ResNet-18/CIFAR-10, beta-scheduling delivers 1.9x faster convergence to 90% accuracy compared to constant momentum.
  arXiv  Detail & Related papers  (2026-03-30T18:53:03Z)
• CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training [73.46600457802693]
  We introduce a new method that counteracts the loss induced by quantization.<n>CAGE significantly improves upon the state-of-theart methods in terms of accuracy, for similar computational cost.<n>For QAT pre-training of Llama models, CAGE matches the accuracy achieved at 4-bits (W4A4) with the prior best method.
  arXiv  Detail & Related papers  (2025-10-21T16:33:57Z)
• Purge-Gate: Backpropagation-Free Test-Time Adaptation for Point Clouds Classification via Token Purging [33.86980091897939]
  Token Purging (PG) is a backpropagation-free approach that removes tokens highly affected by domain shifts before they reach attention layers.<n>We propose two variants: PG-SP, which leverages source statistics, and PG-SF, a fully source-free version relying on CLS-token-driven adaptation.<n>PG-SP achieves an average of +10.3% higher accuracy than state-of-the-art backpropagation-free methods, while PG-SF sets new benchmarks for source-free adaptation.
  arXiv  Detail & Related papers  (2025-09-11T18:33:40Z)
• OrthoGrad Improves Neural Calibration [0.0]
  $perp$Grad constrains descent directions to address overconfidence.<n>$perp$Grad is a geometry-aware modification to optimization.
  arXiv  Detail & Related papers  (2025-06-04T22:12:46Z)
• Decentralized Nonconvex Composite Federated Learning with Gradient Tracking and Momentum [78.27945336558987]
  Decentralized server (DFL) eliminates reliance on client-client architecture.<n>Non-smooth regularization is often incorporated into machine learning tasks.<n>We propose a novel novel DNCFL algorithm to solve these problems.
  arXiv  Detail & Related papers  (2025-04-17T08:32:25Z)
• Exploiting Adam-like Optimization Algorithms to Improve the Performance of Convolutional Neural Networks [82.61182037130405]
  gradient descent (SGD) is the main approach for training deep networks. In this work, we compare Adam based variants based on the difference between the present and the past gradients. We have tested ensemble of networks and the fusion with ResNet50 trained with gradient descent.
  arXiv  Detail & Related papers  (2021-03-26T18:55:08Z)
• Balancing Rates and Variance via Adaptive Batch-Size for Stochastic Optimization Problems [120.21685755278509]
  In this work, we seek to balance the fact that attenuating step-size is required for exact convergence with the fact that constant step-size learns faster in time up to an error. Rather than fixing the minibatch the step-size at the outset, we propose to allow parameters to evolve adaptively.
  arXiv  Detail & Related papers  (2020-07-02T16:02:02Z)
• MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients [112.00379151834242]
  We propose adaptive learning rate principle, in which the running mean of squared gradient in Adam is replaced by a weighted mean, with weights chosen to maximize the estimated variance each coordinate. This results in faster adaptation, which leads more desirable empirical convergence behaviors.
  arXiv  Detail & Related papers  (2020-06-21T21:47:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.