AMStraMGRAM: Adaptive Multi-cutoff Strategy Modification for ANaGRAM

• URL: http://arxiv.org/abs/2510.15998v1
• Date: Tue, 14 Oct 2025 09:10:42 GMT
• Title: AMStraMGRAM: Adaptive Multi-cutoff Strategy Modification for ANaGRAM
• Authors: Nilo Schwencke, Cyriaque Rousselot, Alena Shilova, Cyril Furtlehner,
• Abstract summary: We analyze the training dynamics of PINNs optimized with ANaGRAM.<n>We propose a multi-cutoff adaptation strategy that further enhances ANaGRAM's performance.
• Score: 6.515592049126884
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent works have shown that natural gradient methods can significantly outperform standard optimizers when training physics-informed neural networks (PINNs). In this paper, we analyze the training dynamics of PINNs optimized with ANaGRAM, a natural-gradient-inspired approach employing singular value decomposition with cutoff regularization. Building on this analysis, we propose a multi-cutoff adaptation strategy that further enhances ANaGRAM's performance. Experiments on benchmark PDEs validate the effectiveness of our method, which allows to reach machine precision on some experiments. To provide theoretical grounding, we develop a framework based on spectral theory that explains the necessity of regularization and extend previous shown connections with Green's functions theory.

Related papers

• OBLR-PO: A Theoretical Framework for Stable Reinforcement Learning [12.77713716713937]
  We provide a unified theoretical framework that characterizes the statistical properties of commonly used policy-gradient estimators.<n>We derive an adaptive learning-rate schedule governed by the signal-to-noise ratio (SNR) of gradients.<n>We further show that the variance-optimal baseline is a gradient-weighted estimator, offering a new principle for variance reduction.
  arXiv  Detail & Related papers  (2025-11-28T16:09:28Z)
• Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections [65.36449542323277]
  We present a unified theoretical framework bridgingSupervised Fine-Tuning (SFT) and preference learning in Large Language Model (LLM) post-training.<n>We propose a simple yet effective learning rate reduction approach that yields significant performance improvements.
  arXiv  Detail & Related papers  (2025-06-15T05:42:29Z)
• Solving Nonlinear PDEs with Sparse Radial Basis Function Networks [0.0]
  We propose a novel framework for solving nonlinear PDEs using sparse radial basis function (RBF) networks.<n>This work is motivated by longstanding challenges in traditional RBF collocation methods, along with the limitations of physics-informed neural networks (PINNs) and Gaussian process (GP) approaches.
  arXiv  Detail & Related papers  (2025-05-12T17:12:53Z)
• Gradient Alignment in Physics-informed Neural Networks: A Second-Order Optimization Perspective [12.91773326430686]
  We present theoretical and practical approaches for addressing directional conflicts between loss terms.<n>We show how these conflicts limit first-order methods and show that second-order optimization naturally resolves them.<n>We prove that SOAP, a recently proposed quasi-Newton method, efficiently approximates the Hessian preconditioner.
  arXiv  Detail & Related papers  (2025-02-02T00:21:45Z)
• See Further for Parameter Efficient Fine-tuning by Standing on the Shoulders of Decomposition [56.87609859444084]
  parameter-efficient fine-tuning (PEFT) focuses on optimizing a select subset of parameters while keeping the rest fixed, significantly lowering computational and storage overheads.<n>We take the first step to unify all approaches by dissecting them from a decomposition perspective.<n>We introduce two novel PEFT methods alongside a simple yet effective framework designed to enhance the performance of PEFT techniques across various applications.
  arXiv  Detail & Related papers  (2024-07-07T15:44:42Z)
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint [56.74058752955209]
  This paper studies the alignment process of generative models with Reinforcement Learning from Human Feedback (RLHF) We first identify the primary challenges of existing popular methods like offline PPO and offline DPO as lacking in strategical exploration of the environment. We propose efficient algorithms with finite-sample theoretical guarantees.
  arXiv  Detail & Related papers  (2023-12-18T18:58:42Z)
• Model-Based Reparameterization Policy Gradient Methods: Theory and Practical Algorithms [88.74308282658133]
  Reization (RP) Policy Gradient Methods (PGMs) have been widely adopted for continuous control tasks in robotics and computer graphics. Recent studies have revealed that, when applied to long-term reinforcement learning problems, model-based RP PGMs may experience chaotic and non-smooth optimization landscapes. We propose a spectral normalization method to mitigate the exploding variance issue caused by long model unrolls.
  arXiv  Detail & Related papers  (2023-10-30T18:43:21Z)
• Provable Reward-Agnostic Preference-Based Reinforcement Learning [61.39541986848391]
  Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories. We propose a theoretical reward-agnostic PbRL framework where exploratory trajectories that enable accurate learning of hidden reward functions are acquired.
  arXiv  Detail & Related papers  (2023-05-29T15:00:09Z)
• A theoretical and empirical study of new adaptive algorithms with additional momentum steps and shifted updates for stochastic non-convex optimization [0.0]
  It is thought that adaptive optimization algorithms represent the key pillar behind the of the Learning field. In this paper we introduce adaptive momentum techniques for different non-smooth objective problems.
  arXiv  Detail & Related papers  (2021-10-16T09:47:57Z)
• A Differential Game Theoretic Neural Optimizer for Training Residual Networks [29.82841891919951]
  We propose a generalized Differential Dynamic Programming (DDP) neural architecture that accepts both residual connections and convolution layers. The resulting optimal control representation admits a gameoretic perspective, in which training residual networks can be interpreted as cooperative trajectory optimization on state-augmented systems.
  arXiv  Detail & Related papers  (2020-07-17T10:19:17Z)
• A Theoretical Framework for Target Propagation [75.52598682467817]
  We analyze target propagation (TP), a popular but not yet fully understood alternative to backpropagation (BP) Our theory shows that TP is closely related to Gauss-Newton optimization and thus substantially differs from BP. We provide a first solution to this problem through a novel reconstruction loss that improves feedback weight training.
  arXiv  Detail & Related papers  (2020-06-25T12:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.