Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Size

• URL: http://arxiv.org/abs/2502.11467v1
• Date: Mon, 17 Feb 2025 05:56:11 GMT
• Title: Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Size
• Authors: Naoki Takeshita, Masaaki Imaizumi,
• Abstract summary: Transformers are a type of neural network that have demonstrated remarkable performance across various domains. In this study, we investigate the ability of transformers to approximate column-symmetrics.
• Score: 6.9060054915724
• License:
• Abstract: Transformers are a type of neural network that have demonstrated remarkable performance across various domains, particularly in natural language processing tasks. Motivated by this success, research on the theoretical understanding of transformers has garnered significant attention. A notable example is the mathematical analysis of their approximation power, which validates the empirical expressive capability of transformers. In this study, we investigate the ability of transformers to approximate column-symmetric polynomials, an extension of symmetric polynomials that take matrices as input. Consequently, we establish an explicit relationship between the size of the transformer network and its approximation capability, leveraging the parameter efficiency of transformers and their compatibility with symmetry by focusing on the algebraic properties of symmetric polynomials.

Related papers

• Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion [43.299430093251736]
  We introduce rotation symmetry, a novel form of parameter space symmetry for transformers. Unlike permutation symmetry, rotation symmetry operates in a continuous domain, thereby significantly expanding the equivalence set for transformers. We propose a theoretically optimal matching algorithm as a plug-and-play module to enhance model fusion.
  arXiv  Detail & Related papers  (2025-02-01T01:44:55Z)
• Equivariant Graph Network Approximations of High-Degree Polynomials for Force Field Prediction [62.05532524197309]
  equivariant deep models have shown promise in accurately predicting atomic potentials and force fields in molecular dynamics simulations. In this work, we analyze the equivariant functions for equivariant architecture, and introduce a novel equivariant network, named PACE. As experimented in commonly used benchmarks, PACE demonstrates state-of-the-art performance in predicting atomic energy and force fields.
  arXiv  Detail & Related papers  (2024-11-06T19:34:40Z)
• EulerFormer: Sequential User Behavior Modeling with Complex Vector Attention [88.45459681677369]
  We propose a novel transformer variant with complex vector attention, named EulerFormer. It provides a unified theoretical framework to formulate both semantic difference and positional difference. It is more robust to semantic variations and possesses moresuperior theoretical properties in principle.
  arXiv  Detail & Related papers  (2024-03-26T14:18:43Z)
• AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures [80.28359222380733]
  We design a novel transformer framework, dubbed AlgoFormer, to empower transformers with algorithmic capabilities. In particular, inspired by the structure of human-designed learning algorithms, our transformer framework consists of a pre-transformer that is responsible for task preprocessing. Some theoretical and empirical results are presented to show that the designed transformer has the potential to perform algorithm representation and learning.
  arXiv  Detail & Related papers  (2024-02-21T07:07:54Z)
• Transformers, parallel computation, and logarithmic depth [33.659870765923884]
  We show that a constant number of self-attention layers can efficiently simulate, and be simulated by, a constant number of communication rounds of Massively Parallel Computation.
  arXiv  Detail & Related papers  (2024-02-14T15:54:55Z)
• Towards Understanding Inductive Bias in Transformers: A View From Infinity [9.00214539845063]
  We argue transformers tend to be biased towards more permutation symmetric functions in sequence space. We show that the representation theory of the symmetric group can be used to give quantitative analytical predictions. We argue WikiText dataset, does indeed possess a degree of permutation symmetry.
  arXiv  Detail & Related papers  (2024-02-07T19:00:01Z)
• Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modeling [10.246977481606427]
  We study the mechanisms through which different components of Transformer, such as the dot-product self-attention, affect its expressive power. Our study reveals the roles of critical parameters in the Transformer, such as the number of layers and the number of attention heads.
  arXiv  Detail & Related papers  (2024-02-01T11:43:13Z)
• Approximation and Estimation Ability of Transformers for Sequence-to-Sequence Functions with Infinite Dimensional Input [50.83356836818667]
  We study the approximation and estimation ability of Transformers as sequence-to-sequence functions with infinite dimensional inputs. Our theoretical results support the practical success of Transformers for high dimensional data.
  arXiv  Detail & Related papers  (2023-05-30T02:44:49Z)
• Your Transformer May Not be as Powerful as You Expect [88.11364619182773]
  We mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. We present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. We develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions.
  arXiv  Detail & Related papers  (2022-05-26T14:51:30Z)
• Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks [126.33843752332139]
  We introduce Group-wise Transformation towards a universal yet lightweight Transformer for vision-and-language tasks, termed as LW-Transformer. We apply LW-Transformer to a set of Transformer-based networks, and quantitatively measure them on three vision-and-language tasks and six benchmark datasets. Experimental results show that while saving a large number of parameters and computations, LW-Transformer achieves very competitive performance against the original Transformer networks for vision-and-language tasks.
  arXiv  Detail & Related papers  (2022-04-16T11:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.