Related papers: Tensor networks and efficient descriptions of classical data

Tensor networks and efficient descriptions of classical data

URL: http://arxiv.org/abs/2103.06872v1
Date: Thu, 11 Mar 2021 18:57:16 GMT
Title: Tensor networks and efficient descriptions of classical data
Authors: Sirui Lu, M\'arton Kan\'asz-Nagy, Ivan Kukuljan, J. Ignacio Cirac
Abstract summary: We study how the mutual information between a subregion and its complement scales with the subsystem size $L$. We find that for text, the mutual information scales as a power law $Lnu$ with a close to volume law exponent. For images, the scaling is close to an area law, hinting at 2D tensor networks such as PEPS could have an adequate expressibility.
Score: 0.9176056742068814
License: http://creativecommons.org/licenses/by/4.0/
Abstract: We investigate the potential of tensor network based machine learning methods to scale to large image and text data sets. For that, we study how the mutual information between a subregion and its complement scales with the subsystem size $L$, similarly to how it is done in quantum many-body physics. We find that for text, the mutual information scales as a power law $L^\nu$ with a close to volume law exponent, indicating that text cannot be efficiently described by 1D tensor networks. For images, the scaling is close to an area law, hinting at 2D tensor networks such as PEPS could have an adequate expressibility. For the numerical analysis, we introduce a mutual information estimator based on autoregressive networks, and we also use convolutional neural networks in a neural estimator method.

Related papers

Assessing Neural Network Representations During Training Using Noise-Resilient Diffusion Spectral Entropy [55.014926694758195]
Entropy and mutual information in neural networks provide rich information on the learning process. We leverage data geometry to access the underlying manifold and reliably compute these information-theoretic measures. We show that they form noise-resistant measures of intrinsic dimensionality and relationship strength in high-dimensional simulated data.
arXiv Detail & Related papers (2023-12-04T01:32:42Z)
Diffused Redundancy in Pre-trained Representations [98.55546694886819]
We take a closer look at how features are encoded in pre-trained representations. We find that learned representations in a given layer exhibit a degree of diffuse redundancy. Our findings shed light on the nature of representations learned by pre-trained deep neural networks.
arXiv Detail & Related papers (2023-05-31T21:00:50Z)
Dive into Layers: Neural Network Capacity Bounding using Algebraic Geometry [55.57953219617467]
We show that the learnability of a neural network is directly related to its size. We use Betti numbers to measure the topological geometric complexity of input data and the neural network. We perform the experiments on a real-world dataset MNIST and the results verify our analysis and conclusion.
arXiv Detail & Related papers (2021-09-03T11:45:51Z)
Leveraging Sparse Linear Layers for Debuggable Deep Networks [86.94586860037049]
We show how fitting sparse linear models over learned deep feature representations can lead to more debuggable neural networks. The resulting sparse explanations can help to identify spurious correlations, explain misclassifications, and diagnose model biases in vision and language tasks.
arXiv Detail & Related papers (2021-05-11T08:15:25Z)
Computing the Information Content of Trained Neural Networks [46.34988166338264]
How can neural networks with vastly more weights than training data still generalise? This paper derives both a consistent estimator and a closed-form upper bound on the information content of infinitely wide neural networks.
arXiv Detail & Related papers (2021-03-01T14:38:25Z)
Mutual Information Scaling for Tensor Network Machine Learning [0.0]
We show how a related correlation analysis can be applied to tensor network machine learning. We explore whether classical data possess correlation scaling patterns similar to those found in quantum states. We characterize the scaling patterns in the MNIST and Tiny Images datasets, and find clear evidence of boundary-law scaling in the latter.
arXiv Detail & Related papers (2021-02-27T02:17:51Z)
The Connection Between Approximation, Depth Separation and Learnability in Neural Networks [70.55686685872008]
We study the connection between learnability and approximation capacity. We show that learnability with deep networks of a target function depends on the ability of simpler classes to approximate the target.
arXiv Detail & Related papers (2021-01-31T11:32:30Z)
Tensor-Train Networks for Learning Predictive Modeling of Multidimensional Data [0.0]
A promising strategy is based on tensor networks, which have been very successful in physical and chemical applications. We show that the weights of a multidimensional regression model can be learned by means of tensor networks with the aim of performing a powerful compact representation. An algorithm based on alternating least squares has been proposed for approximating the weights in TT-format with a reduction of computational power.
arXiv Detail & Related papers (2021-01-22T16:14:38Z)
A Greedy Algorithm for Quantizing Neural Networks [4.683806391173103]
We propose a new computationally efficient method for quantizing the weights of pre- trained neural networks. Our method deterministically quantizes layers in an iterative fashion with no complicated re-training required.
arXiv Detail & Related papers (2020-10-29T22:53:10Z)
Tensor Networks for Medical Image Classification [0.456877715768796]
We focus on the class of Networks, which has been a work horse for physicists in the last two decades to analyse quantum many-body systems. We extend the Matrix Product State tensor networks to be useful in medical image analysis tasks. We show that tensor networks are capable of attaining performance that is comparable to state-of-the-art deep learning methods.
arXiv Detail & Related papers (2020-04-21T15:02:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.