Hybrid Least Squares/Gradient Descent Methods for DeepONets

• URL: http://arxiv.org/abs/2508.15394v1
• Date: Thu, 21 Aug 2025 09:34:06 GMT
• Title: Hybrid Least Squares/Gradient Descent Methods for DeepONets
• Abstract summary: The output of DeepONet can be viewed as linear with respect to the last parameters of the branch network.<n>We propose an efficient hybrid least/ient descent method to accelerate DeepONet.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We propose an efficient hybrid least squares/gradient descent method to accelerate DeepONet training. Since the output of DeepONet can be viewed as linear with respect to the last layer parameters of the branch network, these parameters can be optimized using a least squares (LS) solve, and the remaining hidden layer parameters are updated by means of gradient descent form. However, building the LS system for all possible combinations of branch and trunk inputs yields a prohibitively large linear problem that is infeasible to solve directly. To address this issue, our method decomposes the large LS system into two smaller, more manageable subproblems $\unicode{x2014}$ one for the branch network and one for the trunk network $\unicode{x2014}$ and solves them separately. This method is generalized to a broader type of $L^2$ loss with a regularization term for the last layer parameters, including the case of unsupervised learning with physics-informed loss.

Related papers

• Hybrid Least Squares/Gradient Descent Methods for MIONets [2.2940141855172036]
  We propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training.<n>This method generalizes the LSGD method for DeepONets.<n>Our method is compatible with a general type of $L2$ loss with regularization terms for the last layer parameters of each branch.
  arXiv  Detail & Related papers  (2026-07-08T03:59:25Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• A Schrödinger Eigenfunction Method for Long-Horizon Stochastic Optimal Control [25.99003754170971]
  High-dimensional optimal control (SOC) becomes harder with longer planning horizons.<n>We prove that, under the gradient drift assumption, $mathcalL$ is unitarily equivalent to a Schrdinger operator $mathcalS = -+ mathcalV$ with purely discrete spectrum.<n>We evaluate our method on several long-horizon benchmarks, achieving an order-of-magnitude improvement in control accuracy compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-03-24T13:15:48Z)
• MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on Large Language Models [53.36415620647177]
  Semi-structured sparsity offers a promising solution by strategically retaining $N$ elements out of every $M$ weights.<n>Existing (N:M)-compatible approaches typically fall into two categories: rule-based layerwise greedy search, which suffers from considerable errors, and gradient-driven learning, which incurs prohibitive training costs.<n>We propose a novel linear-space probabilistic framework named MaskPro, which aims to learn a prior categorical distribution for every $M$ consecutive weights and subsequently leverages this distribution to generate the (N:M)-sparsity throughout an $N$-way sampling
  arXiv  Detail & Related papers  (2025-06-15T15:02:59Z)
• Find A Winning Sign: Sign Is All We Need to Win the Lottery [52.63674911541416]
  We show that a sparse network trained by an existing IP method can retain its basin of attraction if its parameter signs and normalization layer parameters are preserved.<n>To take a step closer to finding a winning ticket, we alleviate the reliance on normalization layer parameters by preventing high error barriers along the linear path between the sparse network trained by our method and its counterpart with normalization layer parameters.
  arXiv  Detail & Related papers  (2025-04-07T09:30:38Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• Matching the Statistical Query Lower Bound for $k$-Sparse Parity Problems with Sign Stochastic Gradient Descent [83.85536329832722]
  We solve the $k$-sparse parity problem with sign gradient descent (SGD) on two-layer fully-connected neural networks.<n>We show that this approach can efficiently solve the $k$-sparse parity problem on a $d$-dimensional hypercube.<n>We then demonstrate how a trained neural network with sign SGD can effectively approximate this good network, solving the $k$-parity problem with small statistical errors.
  arXiv  Detail & Related papers  (2024-04-18T17:57:53Z)
• Universal Online Learning with Gradient Variations: A Multi-layer Online Ensemble Approach [57.92727189589498]
  We propose an online convex optimization approach with two different levels of adaptivity. We obtain $mathcalO(log V_T)$, $mathcalO(d log V_T)$ and $hatmathcalO(sqrtV_T)$ regret bounds for strongly convex, exp-concave and convex loss functions.
  arXiv  Detail & Related papers  (2023-07-17T09:55:35Z)
• Differentiating Through Integer Linear Programs with Quadratic Regularization and Davis-Yin Splitting [5.199570417938866]
  We study the case where the problem in question is an Linear Program (ILP) We prove that the resulting scheme is compatible with the recently introduced Jacobian-free backpropagation (JFB) Our experiments on two representative ILPs, the shortest path problem and the knapsack problem, demonstrate that this combination-DYS on the forward pass, JFB on the backward pass-yields a scheme which scales more effectively to high-dimensional problems than existing schemes.
  arXiv  Detail & Related papers  (2023-01-31T04:03:28Z)
• Direct Parameterization of Lipschitz-Bounded Deep Networks [3.883460584034766]
  This paper introduces a new parameterization of deep neural networks (both fully-connected and convolutional) with guaranteed $ell2$ Lipschitz bounds. The Lipschitz guarantees are equivalent to the tightest-known bounds based on certification via a semidefinite program (SDP) We provide a direct'' parameterization, i.e., a smooth mapping from $mathbb RN$ onto the set of weights satisfying the SDP-based bound.
  arXiv  Detail & Related papers  (2023-01-27T04:06:31Z)
• Gradient Descent Optimizes Infinite-Depth ReLU Implicit Networks with Linear Widths [25.237054775800164]
  This paper studies the convergence of gradient flow and gradient descent for nonlinear ReLU activated implicit networks. We prove that both GF and GD converge to a global minimum at a linear rate if the width $m$ of the implicit network is textitlinear in the sample size.
  arXiv  Detail & Related papers  (2022-05-16T06:07:56Z)
• Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Bias [34.81794649454105]
  Real-life neural networks are from small random values and trained with cross-entropy loss for classification. Recent results show that gradient descent converges to the "max-margin" solution with zero loss, which presumably generalizes well. The current paper is able to establish this global optimality for two-layer ReLU nets trained with gradient flow on linearly separable and symmetric data.
  arXiv  Detail & Related papers  (2021-10-26T17:57:57Z)
• Self Sparse Generative Adversarial Networks [73.590634413751]
  Generative Adversarial Networks (GANs) are an unsupervised generative model that learns data distribution through adversarial training. We propose a Self Sparse Generative Adversarial Network (Self-Sparse GAN) that reduces the parameter space and alleviates the zero gradient problem.
  arXiv  Detail & Related papers  (2021-01-26T04:49:12Z)
• Global Convergence of Deep Networks with One Wide Layer Followed by Pyramidal Topology [28.49901662584467]
  We prove that, for deep networks, a single layer of width $N$ following the input layer suffices to ensure a similar guarantee. All the remaining layers are allowed to have constant widths, and form a pyramidal topology.
  arXiv  Detail & Related papers  (2020-02-18T20:21:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.