Hybrid Least Squares/Gradient Descent Methods for MIONets

• URL: http://arxiv.org/abs/2607.06976v1
• Date: Wed, 08 Jul 2026 03:59:25 GMT
• Title: Hybrid Least Squares/Gradient Descent Methods for MIONets
• Abstract summary: We propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training.<n>This method generalizes the LSGD method for DeepONets.<n>Our method is compatible with a general type of $L2$ loss with regularization terms for the last layer parameters of each branch.
• Score: 2.2940141855172036
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In this paper, we propose an efficient hybrid least squares/gradient descent (LSGD) method for MIONets to accelerate training. This method generalizes the LSGD method for DeepONets. Since MIONet is the sum of the entrywise product of multiple branch networks and a trunk network, it can be viewed as a multilinear function with respect to the last layer parameters of each branch network. These sets of parameters can be optimized using the alternating least squares method, where we solve the LS system for a single branch network in turn. To handle the large-sized system matrix, we introduce Kronecker and Khatri-Rao products and tensor permutation matrices to factor the large matrix into small ones. Our method is compatible with a general type of $L^2$ loss with regularization terms for the last layer parameters of each branch, where linear operators can be applied to the MIONet output in each loss term.

Related papers

• A fast direct solver based neural network for solving PDEs [0.2578242050187029]
  We present a neural network that learns the inverse operation of HODLR matrices based on the fast direct solver for HODLR matrices developed by Ambikasaran and Darve (2013).<n>We extend the architecture to learn nonlinear solution operators associated with PDEs by replacing some of the linear layers with deep sub-networks.
  arXiv  Detail & Related papers  (2026-06-18T07:51:45Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Hybrid Least Squares/Gradient Descent Methods for DeepONets [0.0]
  The output of DeepONet can be viewed as linear with respect to the last parameters of the branch network.<n>We propose an efficient hybrid least/ient descent method to accelerate DeepONet.
  arXiv  Detail & Related papers  (2025-08-21T09:34:06Z)
• Incorporating Arbitrary Matrix Group Equivariance into KANs [69.30866522377694]
  Kolmogorov-Arnold Networks (KANs) have seen great success in scientific domains.<n>We propose Equivariant Kolmogorov-Arnold Networks (EKAN) to broaden their applicability to more fields.
  arXiv  Detail & Related papers  (2024-10-01T06:34:58Z)
• Layer-Specific Optimization: Sensitivity Based Convolution Layers Basis Search [0.0]
  We propose a new way of applying the matrix decomposition with respect to the weights of convolutional layers. The essence of the method is to train not all convolutions, but only the subset of convolutions (basis convolutions) and represent the rest as linear combinations of the basis ones. Experiments on models from the ResNet family and the CIFAR-10 dataset demonstrate that basis convolutions can not only reduce the size of the model but also accelerate the forward and backward passes of the network.
  arXiv  Detail & Related papers  (2024-08-12T09:24:48Z)
• Over-the-Air Split Machine Learning in Wireless MIMO Networks [56.27831295707334]
  In split machine learning (ML), different partitions of a neural network (NN) are executed by different computing nodes. To ease communication burden, over-the-air computation (OAC) can efficiently implement all or part of the computation at the same time of communication.
  arXiv  Detail & Related papers  (2022-10-07T15:39:11Z)
• High-Dimensional Sparse Bayesian Learning without Covariance Matrices [66.60078365202867]
  We introduce a new inference scheme that avoids explicit construction of the covariance matrix. Our approach couples a little-known diagonal estimation result from numerical linear algebra with the conjugate gradient algorithm. On several simulations, our method scales better than existing approaches in computation time and memory.
  arXiv  Detail & Related papers  (2022-02-25T16:35:26Z)
• A Mini-Block Natural Gradient Method for Deep Neural Networks [12.48022619079224]
  We propose and analyze the convergence of an approximate natural gradient method, mini-block Fisher (MBF) Our novel approach utilizes the parallelism of generalization to efficiently perform on the large number of matrices in each layer.
  arXiv  Detail & Related papers  (2022-02-08T20:01:48Z)
• Memory-Efficient Backpropagation through Large Linear Layers [107.20037639738433]
  In modern neural networks like Transformers, linear layers require significant memory to store activations during backward pass. This study proposes a memory reduction approach to perform backpropagation through linear layers.
  arXiv  Detail & Related papers  (2022-01-31T13:02:41Z)
• Training Recurrent Neural Networks by Sequential Least Squares and the Alternating Direction Method of Multipliers [0.20305676256390928]
  We propose the use of convex and twice-differentiable loss and regularization terms for determining optimal hidden network parameters. We combine sequential least squares with alternating direction multipliers. The performance of the algorithm is tested in a nonlinear system identification benchmark.
  arXiv  Detail & Related papers  (2021-12-31T08:43:04Z)
• Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization [59.87663954467815]
  Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer. In this paper, leveraging a recent linear algebraic theorem called Gershgorin disc perfect alignment (GDPA), we unroll a projection-free algorithm for semi-definite programming relaxation (SDR) of a binary graph. Experimental results show that our unrolled network outperformed pure model-based graph classifiers, and achieved comparable performance to pure data-driven networks but using far fewer parameters.
  arXiv  Detail & Related papers  (2021-09-10T07:01:15Z)
• Controllable Orthogonalization in Training DNNs [96.1365404059924]
  Orthogonality is widely used for training deep neural networks (DNNs) due to its ability to maintain all singular values of the Jacobian close to 1. This paper proposes a computationally efficient and numerically stable orthogonalization method using Newton's iteration (ONI) We show that our method improves the performance of image classification networks by effectively controlling the orthogonality to provide an optimal tradeoff between optimization benefits and representational capacity reduction. We also show that ONI stabilizes the training of generative adversarial networks (GANs) by maintaining the Lipschitz continuity of a network, similar to spectral normalization (
  arXiv  Detail & Related papers  (2020-04-02T10:14:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.