Syntactic Structure from Deep Learning

• URL: http://arxiv.org/abs/2004.10827v1
• Date: Wed, 22 Apr 2020 20:02:49 GMT
• Title: Syntactic Structure from Deep Learning
• Authors: Tal Linzen and Marco Baroni
• Abstract summary: Modern deep neural networks achieve impressive performance in engineering applications that require extensive linguistic skills, such as machine translation. This success has sparked interest in probing whether these models are inducing human-like grammatical knowledge from the raw data they are exposed to.
• Score: 40.84740599926241
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern deep neural networks achieve impressive performance in engineering applications that require extensive linguistic skills, such as machine translation. This success has sparked interest in probing whether these models are inducing human-like grammatical knowledge from the raw data they are exposed to, and, consequently, whether they can shed new light on long-standing debates concerning the innate structure necessary for language acquisition. In this article, we survey representative studies of the syntactic abilities of deep networks, and discuss the broader implications that this work has for theoretical linguistics.

Related papers

• IOLBENCH: Benchmarking LLMs on Linguistic Reasoning [8.20398036986024]
  We introduce IOLBENCH, a novel benchmark derived from International Linguistics Olympiad (IOL) problems. This dataset encompasses diverse problems testing syntax, morphology, phonology, and semantics. We find that even the most advanced models struggle to handle the intricacies of linguistic complexity.
  arXiv  Detail & Related papers  (2025-01-08T03:15:10Z)
• Developmental Predictive Coding Model for Early Infancy Mono and Bilingual Vocal Continual Learning [69.8008228833895]
  We propose a small-sized generative neural network equipped with a continual learning mechanism. Our model prioritizes interpretability and demonstrates the advantages of online learning.
  arXiv  Detail & Related papers  (2024-12-23T10:23:47Z)
• Analysis and Visualization of Linguistic Structures in Large Language Models: Neural Representations of Verb-Particle Constructions in BERT [0.0]
  This study investigates the internal representations of verb-particle combinations within large language models (LLMs) We analyse the representational efficacy of its layers for various verb-particle constructions such as 'agree on', 'come back', and 'give up' Results show that BERT's middle layers most effectively capture syntactic structures, with significant variability in representational accuracy across different verb categories.
  arXiv  Detail & Related papers  (2024-12-19T09:21:39Z)
• Finding Structure in Language Models [3.882018118763685]
  This thesis is about whether language models possess a deep understanding of grammatical structure similar to that of humans. We will develop novel interpretability techniques that enhance our understanding of the complex nature of large-scale language models.
  arXiv  Detail & Related papers  (2024-11-25T14:37:24Z)
• Language Models as Models of Language [0.0]
  This chapter critically examines the potential contributions of modern language models to theoretical linguistics. I review a growing body of empirical evidence suggesting that language models can learn hierarchical syntactic structure and exhibit sensitivity to various linguistic phenomena. I conclude that closer collaboration between theoretical linguists and computational researchers could yield valuable insights.
  arXiv  Detail & Related papers  (2024-08-13T18:26:04Z)
• SINC: Self-Supervised In-Context Learning for Vision-Language Tasks [64.44336003123102]
  We propose a framework to enable in-context learning in large language models. A meta-model can learn on self-supervised prompts consisting of tailored demonstrations. Experiments show that SINC outperforms gradient-based methods in various vision-language tasks.
  arXiv  Detail & Related papers  (2023-07-15T08:33:08Z)
• On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex [48.588772371355816]
  This paper presents the first empirical study on the adversarial robustness of a large prompt-based language model of code, codex. Our results demonstrate that the state-of-the-art (SOTA) code-language models are vulnerable to carefully crafted adversarial examples.
  arXiv  Detail & Related papers  (2023-01-30T13:21:00Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• Schr\"odinger's Tree -- On Syntax and Neural Language Models [10.296219074343785]
  Language models have emerged as NLP's workhorse, displaying increasingly fluent generation capabilities. We observe a lack of clarity across numerous dimensions, which influences the hypotheses that researchers form. We outline the implications of the different types of research questions exhibited in studies on syntax.
  arXiv  Detail & Related papers  (2021-10-17T18:25:23Z)
• On the proper role of linguistically-oriented deep net analysis in linguistic theorizing [25.64606911182175]
  I suggest that deep networks should be treated as theories making explicit predictions about the acceptability of linguistic utterances. I argue that, if we overcome some obstacles standing in the way of seriously pursuing this idea, we will gain a powerful new theoretical tool.
  arXiv  Detail & Related papers  (2021-06-16T10:57:24Z)
• Introducing Syntactic Structures into Target Opinion Word Extraction with Deep Learning [89.64620296557177]
  We propose to incorporate the syntactic structures of the sentences into the deep learning models for targeted opinion word extraction. We also introduce a novel regularization technique to improve the performance of the deep learning models. The proposed model is extensively analyzed and achieves the state-of-the-art performance on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-26T07:13:17Z)
• The Unstoppable Rise of Computational Linguistics in Deep Learning [17.572024590374305]
  We trace the history of neural networks applied to natural language understanding tasks. We argue that Transformer is not a sequence model but an induced-structure model.
  arXiv  Detail & Related papers  (2020-05-13T16:51:02Z)
• Information-Theoretic Probing for Linguistic Structure [74.04862204427944]
  We propose an information-theoretic operationalization of probing as estimating mutual information. We evaluate on a set of ten typologically diverse languages often underrepresented in NLP research.
  arXiv  Detail & Related papers  (2020-04-07T01:06:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.