Understanding Deflation Process in Over-parametrized Tensor Decomposition

• URL: http://arxiv.org/abs/2106.06573v1
• Date: Fri, 11 Jun 2021 18:51:36 GMT
• Title: Understanding Deflation Process in Over-parametrized Tensor Decomposition
• Authors: Rong Ge, Yunwei Ren, Xiang Wang, Mo Zhou
• Abstract summary: We study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components.
• Score: 17.28303004783945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.

Related papers

• Implicit Regularization for Tubal Tensor Factorizations via Gradient Descent [4.031100721019478]
  We provide a rigorous analysis of implicit regularization in an overparametrized tensor factorization problem beyond the lazy training regime. We prove the first tensor result of its kind for gradient descent rather than gradient flow.
  arXiv  Detail & Related papers  (2024-10-21T17:52:01Z)
• Tensor cumulants for statistical inference on invariant distributions [49.80012009682584]
  We show that PCA becomes computationally hard at a critical value of the signal's magnitude. We define a new set of objects, which provide an explicit, near-orthogonal basis for invariants of a given degree. It also lets us analyze a new problem of distinguishing between different ensembles.
  arXiv  Detail & Related papers  (2024-04-29T14:33:24Z)
• Mitigating Heterogeneity among Factor Tensors via Lie Group Manifolds for Tensor Decomposition Based Temporal Knowledge Graph Embedding [32.87000154536683]
  We introduce a novel method that maps factor tensors onto a unified smooth Lie group manifold to make the distribution of factor tensors approximating homogeneous in tensor decomposition. The proposed method can be directly integrated into existing tensor decomposition based TKGE methods without introducing extra parameters.
  arXiv  Detail & Related papers  (2024-04-14T06:10:46Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Scalable CP Decomposition for Tensor Learning using GPU Tensor Cores [47.87810316745786]
  We propose a compression-based tensor decomposition framework, namely the exascale-tensor, to support exascale tensor decomposition. Compared to the baselines, the exascale-tensor supports 8,000x larger tensors and a speedup up to 6.95x. We also apply our method to two real-world applications, including gene analysis and tensor layer neural networks.
  arXiv  Detail & Related papers  (2023-11-22T21:04:59Z)
• Provable Tensor Completion with Graph Information [49.08648842312456]
  We introduce a novel model, theory, and algorithm for solving the dynamic graph regularized tensor completion problem. We develop a comprehensive model simultaneously capturing the low-rank and similarity structure of the tensor. In terms of theory, we showcase the alignment between the proposed graph smoothness regularization and a weighted tensor nuclear norm.
  arXiv  Detail & Related papers  (2023-10-04T02:55:10Z)
• Decomposition of linear tensor transformations [0.0]
  The aim of this paper is to develop a mathematical framework for exact tensor decomposition. In the paper three different problems will be carried out to derive.
  arXiv  Detail & Related papers  (2023-09-14T16:14:38Z)
• Optimizing Orthogonalized Tensor Deflation via Random Tensor Theory [5.124256074746721]
  This paper tackles the problem of recovering a low-rank signal tensor with possibly correlated components from a random noisy tensor. Non-orthogonal components may alter the tensor deflation mechanism, thereby preventing efficient recovery. An efficient tensor deflation algorithm is proposed by optimizing the parameter introduced in the deflation mechanism.
  arXiv  Detail & Related papers  (2023-02-11T22:23:27Z)
• Error Analysis of Tensor-Train Cross Approximation [88.83467216606778]
  We provide accuracy guarantees in terms of the entire tensor for both exact and noisy measurements. Results are verified by numerical experiments, and may have important implications for the usefulness of cross approximations for high-order tensors.
  arXiv  Detail & Related papers  (2022-07-09T19:33:59Z)
• MTC: Multiresolution Tensor Completion from Partial and Coarse Observations [49.931849672492305]
  Existing completion formulation mostly relies on partial observations from a single tensor. We propose an efficient Multi-resolution Completion model (MTC) to solve the problem.
  arXiv  Detail & Related papers  (2021-06-14T02:20:03Z)
• Alternating linear scheme in a Bayesian framework for low-rank tensor approximation [5.833272638548154]
  We find a low-rank representation for a given tensor by solving a Bayesian inference problem. We present an algorithm that performs the unscented transform in tensor train format.
  arXiv  Detail & Related papers  (2020-12-21T10:15:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.