Natural Riemannian gradient for learning functional tensor networks

• URL: http://arxiv.org/abs/2604.09263v1
• Date: Fri, 10 Apr 2026 12:25:20 GMT
• Title: Natural Riemannian gradient for learning functional tensor networks
• Abstract summary: We consider machine learning tasks with low-rank functional tree tensor networks (TTN) as the learning model.<n>Our framework applies to both the factorized and manifold-based approach for representing the functional TTN.
• Score: 0.08999666725996974
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider machine learning tasks with low-rank functional tree tensor networks (TTN) as the learning model. While in the case of least-squares regression, low-rank functional TTNs can be efficiently optimized using alternating optimization, this is not directly possible in other problems, such as multinomial logistic regression. We propose a natural Riemannian gradient descent type approach applicable to arbitrary losses which is based on the natural gradient by Amari. In particular, the search direction obtained by the natural gradient is independent of the choice of basis of the underlying functional tensor product space. Our framework applies to both the factorized and manifold-based approach for representing the functional TTN. For practical application, we propose a hierarchy of efficient approximations to the true natural Riemannian gradient for computing the updates in the parameter space. Numerical experiments confirm our theoretical findings on common classification datasets and show that using natural Riemannian gradient descent for learning considerably improves convergence behavior when compared to standard Riemannian gradient methods.

Related papers

• Functional Gradient Descent with Adaptive Representations [6.479603810066888]
  We propose a new, theoretically-grounded functional descent (FGD) algorithm for functional optimization problems.<n>FGD is difficult to implement in practice because functional gradients cannot be fully computed nor stored in memory.<n>We show that our algorithm consistently outperforms both FGD with fixed approximations and neural network baselines in efficiency accuracy.
  arXiv  Detail & Related papers  (2026-06-15T16:27:25Z)
• Natural gradient descent with momentum [0.0]
  We introduce a natural version of classical inertial dynamic methods like Heavy-Ball or Nesterov.<n>This work shows how it can improve the learning process when working with nonlinear model classes.
  arXiv  Detail & Related papers  (2026-04-16T22:09:39Z)
• ONG: Orthogonal Natural Gradient Descent [0.0]
  We introduce the Orthogonal Natural Gradient Descent (ONG) algorithm.<n>ONG preconditions each new task-specific gradient with an efficient EKFAC approximation of the inverse Fisher information matrix.<n>To preserve performance on previously learned tasks, ONG projects these natural gradients onto the complement of prior tasks' gradients.
  arXiv  Detail & Related papers  (2025-08-24T00:27:23Z)
• Mathematical analysis of the gradients in deep learning [3.3123773366516645]
  We show that a gradient function must coincide with the standard gradient of the cost functional on every open sets on which the cost functional is continuously differentiable.<n>We conclude that the generalized gradient function must coincide with the standard gradient of the cost functional on every open sets on which the cost functional is continuously differentiable.
  arXiv  Detail & Related papers  (2025-01-26T19:11:57Z)
• Neural Gradient Learning and Optimization for Oriented Point Normal Estimation [53.611206368815125]
  We propose a deep learning approach to learn gradient vectors with consistent orientation from 3D point clouds for normal estimation. We learn an angular distance field based on local plane geometry to refine the coarse gradient vectors. Our method efficiently conducts global gradient approximation while achieving better accuracy and ability generalization of local feature description.
  arXiv  Detail & Related papers  (2023-09-17T08:35:11Z)
• Stochastic Marginal Likelihood Gradients using Neural Tangent Kernels [78.6096486885658]
  We introduce lower bounds to the linearized Laplace approximation of the marginal likelihood. These bounds are amenable togradient-based optimization and allow to trade off estimation accuracy against computational complexity.
  arXiv  Detail & Related papers  (2023-06-06T19:02:57Z)
• Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization [73.80101701431103]
  The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. We study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility.
  arXiv  Detail & Related papers  (2023-04-17T14:23:43Z)
• Achieving High Accuracy with PINNs via Energy Natural Gradients [0.0]
  We show that the update direction in function space resulting from the energy natural gradient corresponds to the Newton direction modulo an projection onto the model's tangent space. We demonstrate experimentally that energy natural gradient descent yields highly accurate solutions with errors several orders of magnitude smaller than what is obtained when training PINNs with standard gradient descent or Adam.
  arXiv  Detail & Related papers  (2023-02-25T21:17:19Z)
• Tame Riemannian Stochastic Approximation [2.8166046619628315]
  We study the properties of approximation applied to a tame nondifferentiable function subject to constraints defined by a Riemannian manifold.<n>Recent work has shown that this class of functions faithfully model the loss landscape of deep neural network training objectives.
  arXiv  Detail & Related papers  (2023-02-01T19:03:00Z)
• Zeroth-Order Hybrid Gradient Descent: Towards A Principled Black-Box Optimization Framework [100.36569795440889]
  This work is on the iteration of zero-th-order (ZO) optimization which does not require first-order information. We show that with a graceful design in coordinate importance sampling, the proposed ZO optimization method is efficient both in terms of complexity as well as as function query cost.
  arXiv  Detail & Related papers  (2020-12-21T17:29:58Z)
• Sinkhorn Natural Gradient for Generative Models [125.89871274202439]
  We propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence. We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically. In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.
  arXiv  Detail & Related papers  (2020-11-09T02:51:17Z)
• Channel-Directed Gradients for Optimization of Convolutional Neural Networks [50.34913837546743]
  We introduce optimization methods for convolutional neural networks that can be used to improve existing gradient-based optimization in terms of generalization error. We show that defining the gradients along the output channel direction leads to a performance boost, while other directions can be detrimental.
  arXiv  Detail & Related papers  (2020-08-25T00:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.