Permutation Search of Tensor Network Structures via Local Sampling

• URL: http://arxiv.org/abs/2206.06597v1
• Date: Tue, 14 Jun 2022 05:12:49 GMT
• Title: Permutation Search of Tensor Network Structures via Local Sampling
• Authors: Chao Li, Junhua Zeng, Zerui Tao, Qibin Zhao
• Abstract summary: In this paper, we consider a practical variant of TN-SS, dubbed TN permutation search (TN-PS) We propose a practically-efficient algorithm to resolve the problem of TN-PS. Numerical results demonstrate that the new algorithm can reduce the required model size of TNs in extensive benchmarks.
• Score: 27.155329364896144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent works put much effort into tensor network structure search (TN-SS), aiming to select suitable tensor network (TN) structures, involving the TN-ranks, formats, and so on, for the decomposition or learning tasks. In this paper, we consider a practical variant of TN-SS, dubbed TN permutation search (TN-PS), in which we search for good mappings from tensor modes onto TN vertices (core tensors) for compact TN representations. We conduct a theoretical investigation of TN-PS and propose a practically-efficient algorithm to resolve the problem. Theoretically, we prove the counting and metric properties of search spaces of TN-PS, analyzing for the first time the impact of TN structures on these unique properties. Numerically, we propose a novel meta-heuristic algorithm, in which the searching is done by randomly sampling in a neighborhood established in our theory, and then recurrently updating the neighborhood until convergence. Numerical results demonstrate that the new algorithm can reduce the required model size of TNs in extensive benchmarks, implying the improvement in the expressive power of TNs. Furthermore, the computational cost for the new algorithm is significantly less than that in~\cite{li2020evolutionary}.

Related papers

• Automatic Structural Search of Tensor Network States including Entanglement Renormalization [0.0]
  Network states, including entanglement renormalization, can encompass a wider variety of entangled states. A proposal has yet to show a structural search of ER due to its high computational cost and the lack of flexibility in its algorithm. In this study, we conducted an optimal structural search of TN, including ER, based on the reconstruction of their local structures with respect to variational energy.
  arXiv  Detail & Related papers  (2024-05-10T15:24:10Z)
• tnGPS: Discovering Unknown Tensor Network Structure Search Algorithms via Large Language Models (LLMs) [31.69308712027795]
  We study how to harness large language models to automatically discover new tensor network structure search (TN-SS) algorithms. By observing how human experts innovate in research, we propose an automatic algorithm discovery framework called tnGPS. The proposed framework is an elaborate prompting pipeline that instruct LLMs to generate new TN-SS algorithms through iterative refinement and enhancement.
  arXiv  Detail & Related papers  (2024-02-04T12:06:13Z)
• LC-TTFS: Towards Lossless Network Conversion for Spiking Neural Networks with TTFS Coding [55.64533786293656]
  We show that our algorithm can achieve a near-perfect mapping between the activation values of an ANN and the spike times of an SNN on a number of challenging AI tasks. The study paves the way for deploying ultra-low-power TTFS-based SNNs on power-constrained edge computing platforms.
  arXiv  Detail & Related papers  (2023-10-23T14:26:16Z)
• SVDinsTN: A Tensor Network Paradigm for Efficient Structure Search from Regularized Modeling Perspective [41.62808372395741]
  Network (TN) representation is a powerful technique for computer vision and machine learning. TN structure search (TN-SS) aims to search for a customized structure to achieve a compact representation, which is a challenging NP-hard problem. We propose a novel TN paradigm, named SVD-inspired TN decomposition (SVDinsTN), which allows us to efficiently solve the TN-SS problem from a regularized modeling perspective.
  arXiv  Detail & Related papers  (2023-05-24T09:02:01Z)
• Neural Functional Transformers [99.98750156515437]
  This paper uses the attention mechanism to define a novel set of permutation equivariant weight-space layers called neural functional Transformers (NFTs) NFTs respect weight-space permutation symmetries while incorporating the advantages of attention, which have exhibited remarkable success across multiple domains. We also leverage NFTs to develop Inr2Array, a novel method for computing permutation invariant representations from the weights of implicit neural representations (INRs)
  arXiv  Detail & Related papers  (2023-05-22T23:38:27Z)
• Alternating Local Enumeration (TnALE): Solving Tensor Network Structure Search with Fewer Evaluations [24.437786843413697]
  We propose TnALE, a new algorithm that updates each structure-related variable alternately by local enumeration. We show that TnALE can find practically good TN-ranks and permutations with vastly fewer evaluations than the state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2023-04-25T14:45:59Z)
• Random Features for the Neural Tangent Kernel [57.132634274795066]
  We propose an efficient feature map construction of the Neural Tangent Kernel (NTK) of fully-connected ReLU network. We show that dimension of the resulting features is much smaller than other baseline feature map constructions to achieve comparable error bounds both in theory and practice.
  arXiv  Detail & Related papers  (2021-04-03T09:08:12Z)
• Adaptive Learning of Tensor Network Structures [6.407946291544721]
  We leverage the TN formalism to develop a generic and efficient adaptive algorithm to learn the structure and the parameters of a TN from data. Our algorithm can adaptively identify TN structures with small number of parameters that effectively optimize any differentiable objective function.
  arXiv  Detail & Related papers  (2020-08-12T16:41:56Z)
• Tensor-to-Vector Regression for Multi-channel Speech Enhancement based on Tensor-Train Network [53.47564132861866]
  We propose a tensor-to-vector regression approach to multi-channel speech enhancement. The key idea is to cast the conventional deep neural network (DNN) based vector-to-vector regression formulation under a tensor-train network (TTN) framework. In 8-channel conditions, a PESQ of 3.12 is achieved using 20 million parameters for TTN, whereas a DNN with 68 million parameters can only attain a PESQ of 3.06.
  arXiv  Detail & Related papers  (2020-02-03T02:58:00Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.