TreeCoders: Trees of Transformers

• URL: http://arxiv.org/abs/2411.07218v1
• Date: Mon, 11 Nov 2024 18:40:04 GMT
• Title: TreeCoders: Trees of Transformers
• Authors: Pierre Colonna D'Istria, Abdulrahman Altahhan,
• Abstract summary: We introduce TreeCoders, a novel family of transformer trees. Transformers serve as nodes, and generic classifiers learn to select the best child. TreeCoders naturally lends itself to distributed implementation.
• Score: 0.0
• License:
• Abstract: In this paper, we introduce TreeCoders, a novel family of transformer trees. We moved away from traditional linear transformers to complete k-ary trees. Transformer blocks serve as nodes, and generic classifiers learn to select the best child and route the sequence of tokens to a specific leaf. The selectors, moved outside the transformer blocks, allow for the use of a variety of architecture without further modifications. Furthermore, our proposed architecture supports sparse node activation due to the logarithmic complexity of a tree search. We validate our idea by testing a series of decoder-only tree transformers, achieving competitive results across a diverse range of language datasets. Our study demonstrates that the proposed tree transformer model outperforms a size-equivalent linear transformer model 76\% of the time over a wide range of tree architectures. Furthermore, our proposed model naturally lends itself to distributed implementation.

Related papers

• Tree Transformers are an Ineffective Model of Syntactic Constituency [0.0]
  Linguists have long held that a key aspect of natural language syntax is the organization of language units into constituent structures. A number of alternative models have been proposed to provide inductive biases towards constituency, including the Tree Transformer. We investigate Tree Transformers to study whether they utilize meaningful and/or useful constituent structures.
  arXiv  Detail & Related papers  (2024-11-25T23:53:46Z)
• Terminating Differentiable Tree Experts [77.2443883991608]
  We propose a neuro-symbolic Differentiable Tree Machine that learns tree operations using a combination of transformers and Representation Products. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. We additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically.
  arXiv  Detail & Related papers  (2024-07-02T08:45:38Z)
• Transformers Provably Learn Sparse Token Selection While Fully-Connected Nets Cannot [50.16171384920963]
  transformer architecture has prevailed in various deep learning settings. One-layer transformer trained with gradient descent provably learns the sparse token selection task.
  arXiv  Detail & Related papers  (2024-06-11T02:15:53Z)
• iTransformer: Inverted Transformers Are Effective for Time Series Forecasting [62.40166958002558]
  We propose iTransformer, which simply applies the attention and feed-forward network on the inverted dimensions. The iTransformer model achieves state-of-the-art on challenging real-world datasets.
  arXiv  Detail & Related papers  (2023-10-10T13:44:09Z)
• Tree Variational Autoencoders [5.992683455757179]
  We propose a new generative hierarchical clustering model that learns a flexible tree-based posterior distribution over latent variables. TreeVAE hierarchically divides samples according to their intrinsic characteristics, shedding light on hidden structures in the data.
  arXiv  Detail & Related papers  (2023-06-15T09:25:04Z)
• Differentiable Tree Operations Promote Compositional Generalization [106.59434079287661]
  Differentiable Tree Machine (DTM) architecture integrates interpreter with external memory and agent that learns to sequentially select tree operations. DTM achieves 100% while existing baselines such as Transformer, Tree Transformer, LSTM, and Tree2Tree LSTM achieve less than 30%.
  arXiv  Detail & Related papers  (2023-06-01T14:46:34Z)
• Characterizing Intrinsic Compositionality in Transformers with Tree Projections [72.45375959893218]
  neural models like transformers can route information arbitrarily between different parts of their input. We show that transformers for three different tasks become more treelike over the course of training. These trees are predictive of model behavior, with more tree-like models generalizing better on tests of compositional generalization.
  arXiv  Detail & Related papers  (2022-11-02T17:10:07Z)
• Trees in transformers: a theoretical analysis of the Transformer's ability to represent trees [6.576972696596151]
  We first analyze the theoretical capability of the standard Transformer architecture to learn tree structures. This implies that a Transformer can learn tree structures well in theory. We conduct experiments with synthetic data and find that the standard Transformer achieves similar accuracy compared to a Transformer where tree position information is explicitly encoded.
  arXiv  Detail & Related papers  (2021-12-16T00:02:02Z)
• Tree-structured Attention with Hierarchical Accumulation [103.47584968330325]
  "Hierarchical Accumulation" encodes parse tree structures into self-attention at constant time complexity. Our approach outperforms SOTA methods in four IWSLT translation tasks and the WMT'14 English-German translation task.
  arXiv  Detail & Related papers  (2020-02-19T08:17:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.