Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations

• URL: http://arxiv.org/abs/2407.04543v1
• Date: Fri, 5 Jul 2024 14:29:44 GMT
• Title: Strengthening Structural Inductive Biases by Pre-training to Perform Syntactic Transformations
• Authors: Matthias Lindemann, Alexander Koller, Ivan Titov,
• Abstract summary: We propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token.
• Score: 75.14793516745374
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Models need appropriate inductive biases to effectively learn from small amounts of data and generalize systematically outside of the training distribution. While Transformers are highly versatile and powerful, they can still benefit from enhanced structural inductive biases for seq2seq tasks, especially those involving syntactic transformations, such as converting active to passive voice or semantic parsing. In this paper, we propose to strengthen the structural inductive bias of a Transformer by intermediate pre-training to perform synthetically generated syntactic transformations of dependency trees given a description of the transformation. Our experiments confirm that this helps with few-shot learning of syntactic tasks such as chunking, and also improves structural generalization for semantic parsing. Our analysis shows that the intermediate pre-training leads to attention heads that keep track of which syntactic transformation needs to be applied to which token, and that the model can leverage these attention heads on downstream tasks.

Related papers

• A distributional simplicity bias in the learning dynamics of transformers [50.91742043564049]
  We show that transformers, trained on natural language data, also display a simplicity bias. Specifically, they sequentially learn many-body interactions among input tokens, reaching a saturation point in the prediction error for low-degree interactions. This approach opens up the possibilities of studying how interactions of different orders in the data affect learning, in natural language processing and beyond.
  arXiv  Detail & Related papers  (2024-10-25T15:39:34Z)
• Interpreting Affine Recurrence Learning in GPT-style Transformers [54.01174470722201]
  In-context learning allows GPT-style transformers to generalize during inference without modifying their weights. This paper focuses specifically on their ability to learn and predict affine recurrences as an ICL task. We analyze the model's internal operations using both empirical and theoretical approaches.
  arXiv  Detail & Related papers  (2024-10-22T21:30:01Z)
• Learning Syntax Without Planting Trees: Understanding When and Why Transformers Generalize Hierarchically [74.96551626420188]
  Transformers trained on natural language data have been shown to learn its hierarchical structure and generalize to sentences with unseen syntactic structures. We investigate sources of inductive bias in transformer models and their training that could cause such generalization behavior to emerge.
  arXiv  Detail & Related papers  (2024-04-25T07:10:29Z)
• Transduce: learning transduction grammars for string transformation [0.0]
  A new algorithm, Transduce, is proposed to learn positional transformations efficiently from one or two positive examples without inductive bias. We experimentally demonstrate that Transduce can learn positional transformations efficiently from one or two positive examples without inductive bias, achieving a success rate higher than the current state of the art.
  arXiv  Detail & Related papers  (2023-12-14T07:59:02Z)
• SIP: Injecting a Structural Inductive Bias into a Seq2Seq Model by Simulation [75.14793516745374]
  We show how a structural inductive bias can be efficiently injected into a seq2seq model by pre-training it to simulate structural transformations on synthetic data. Our experiments show that our method imparts the desired inductive bias, resulting in better few-shot learning for FST-like tasks.
  arXiv  Detail & Related papers  (2023-10-01T21:19:12Z)
• BayesFormer: Transformer with Uncertainty Estimation [31.206243748162553]
  We introduce BayesFormer, a Transformer model with dropouts designed by Bayesian theory. We show improvements across the board: language modeling and classification, long-sequence understanding, machine translation and acquisition function for active learning.
  arXiv  Detail & Related papers  (2022-06-02T01:54:58Z)
• Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale [31.293175512404172]
  We introduce Transformer Grammars -- a class of Transformer language models that combine expressive power, scalability, and strong performance of Transformers. We find that Transformer Grammars outperform various strong baselines on multiple syntax-sensitive language modeling evaluation metrics.
  arXiv  Detail & Related papers  (2022-03-01T17:22:31Z)
• Pretrained Transformers as Universal Computation Engines [105.00539596788127]
  We investigate the capability of a transformer pretrained on natural language to generalize to other modalities with minimal finetuning. We study finetuning it on a variety of sequence classification tasks spanning numerical computation, vision, and protein fold prediction. We find that such pretraining enables FPT to generalize in zero-shot to these modalities, matching the performance of a transformer fully trained on these tasks.
  arXiv  Detail & Related papers  (2021-03-09T06:39:56Z)
• Do Syntax Trees Help Pre-trained Transformers Extract Information? [8.133145094593502]
  We study the utility of incorporating dependency trees into pre-trained transformers on information extraction tasks. We propose and investigate two distinct strategies for incorporating dependency structure. We find that their performance gains are highly contingent on the availability of human-annotated dependency parses.
  arXiv  Detail & Related papers  (2020-08-20T17:17:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.