Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT

• URL: http://arxiv.org/abs/2209.10538v1
• Date: Wed, 21 Sep 2022 17:57:23 GMT
• Title: Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT
• Authors: Karim Lasri and Olga Seminck and Alessandro Lenci and Thierry Poibeau
• Abstract summary: We test whether meaning interferes with subject-verb number agreement in English. We generate semantically well-formed and nonsensical items. We find that BERT and humans are both sensitive to our semantic manipulation.
• Score: 64.40111510974957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Both humans and neural language models are able to perform subject-verb number agreement (SVA). In principle, semantics shouldn't interfere with this task, which only requires syntactic knowledge. In this work we test whether meaning interferes with this type of agreement in English in syntactic structures of various complexities. To do so, we generate both semantically well-formed and nonsensical items. We compare the performance of BERT-base to that of humans, obtained with a psycholinguistic online crowdsourcing experiment. We find that BERT and humans are both sensitive to our semantic manipulation: They fail more often when presented with nonsensical items, especially when their syntactic structure features an attractor (a noun phrase between the subject and the verb that has not the same number as the subject). We also find that the effect of meaningfulness on SVA errors is stronger for BERT than for humans, showing higher lexical sensitivity of the former on this task.

Related papers

• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis. Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• A Psycholinguistic Analysis of BERT's Representations of Compounds [3.034345346208211]
  We build on studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions. We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis. We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations.
  arXiv  Detail & Related papers  (2023-02-14T18:23:15Z)
• Abstraction not Memory: BERT and the English Article System [1.1064955465386002]
  We compare the performance of native English speakers and pre-trained models on the task of article prediction set up as a three way choice (a/an, the, zero) Our experiments with BERT show that BERT outperforms humans on this task across all articles.
  arXiv  Detail & Related papers  (2022-06-08T22:36:54Z)
• How Does Adversarial Fine-Tuning Benefit BERT? [16.57274211257757]
  Adversarial training is one of the most reliable methods for defending against adversarial attacks in machine learning. We show that adversarially fine-tuned models remain more faithful to BERT's language modeling behavior and are more sensitive to the word order. Our analysis demonstrates that vanilla fine-tuning oversimplifies the sentence representation by focusing heavily on one or a few label-indicative words.
  arXiv  Detail & Related papers  (2021-08-31T03:39:06Z)
• RAW-C: Relatedness of Ambiguous Words--in Context (A New Lexical Resource for English) [2.792030485253753]
  We evaluate how well contextualized embeddings accommodate the continuous, dynamic nature of word meaning. We show that cosine distance systematically underestimates how similar humans find uses of the same sense of a word to be. We propose a synthesis between psycholinguistic theories of the mental lexicon and computational models of lexical semantics.
  arXiv  Detail & Related papers  (2021-05-27T16:07:13Z)
• Towards Abstract Relational Learning in Human Robot Interaction [73.67226556788498]
  Humans have a rich representation of the entities in their environment. If robots need to interact successfully with humans, they need to represent entities, attributes, and generalizations in a similar way. In this work, we address the problem of how to obtain these representations through human-robot interaction.
  arXiv  Detail & Related papers  (2020-11-20T12:06:46Z)
• On the Sentence Embeddings from Pre-trained Language Models [78.45172445684126]
  In this paper, we argue that the semantic information in the BERT embeddings is not fully exploited. We find that BERT always induces a non-smooth anisotropic semantic space of sentences, which harms its performance of semantic similarity. We propose to transform the anisotropic sentence embedding distribution to a smooth and isotropic Gaussian distribution through normalizing flows that are learned with an unsupervised objective.
  arXiv  Detail & Related papers  (2020-11-02T13:14:57Z)
• Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming [8.08493736237816]
  We present a case study analyzing the pre-trained BERT model with tests informed by semantic priming. We find that BERT too shows "priming," predicting a word with greater probability when the context includes a related word versus an unrelated one. Follow-up analysis shows BERT to be increasingly distracted by related prime words as context becomes more informative.
  arXiv  Detail & Related papers  (2020-10-06T20:30:59Z)
• Speakers Fill Lexical Semantic Gaps with Context [65.08205006886591]
  We operationalise the lexical ambiguity of a word as the entropy of meanings it can take. We find significant correlations between our estimate of ambiguity and the number of synonyms a word has in WordNet. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.
  arXiv  Detail & Related papers  (2020-10-05T17:19:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.