A Psycholinguistic Analysis of BERT's Representations of Compounds

• URL: http://arxiv.org/abs/2302.07232v1
• Date: Tue, 14 Feb 2023 18:23:15 GMT
• Title: A Psycholinguistic Analysis of BERT's Representations of Compounds
• Authors: Lars Buijtelaar, Sandro Pezzelle
• Abstract summary: We build on studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions. We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis. We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations.
• Score: 3.034345346208211
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work studies the semantic representations learned by BERT for compounds, that is, expressions such as sunlight or bodyguard. We build on recent studies that explore semantic information in Transformers at the word level and test whether BERT aligns with human semantic intuitions when dealing with expressions (e.g., sunlight) whose overall meaning depends -- to a various extent -- on the semantics of the constituent words (sun, light). We leverage a dataset that includes human judgments on two psycholinguistic measures of compound semantic analysis: lexeme meaning dominance (LMD; quantifying the weight of each constituent toward the compound meaning) and semantic transparency (ST; evaluating the extent to which the compound meaning is recoverable from the constituents' semantics). We show that BERT-based measures moderately align with human intuitions, especially when using contextualized representations, and that LMD is overall more predictable than ST. Contrary to the results reported for 'standard' words, higher, more contextualized layers are the best at representing compound meaning. These findings shed new light on the abilities of BERT in dealing with fine-grained semantic phenomena. Moreover, they can provide insights into how speakers represent compounds.

Related papers

• Syntax and Semantics Meet in the "Middle": Probing the Syntax-Semantics Interface of LMs Through Agentivity [68.8204255655161]
  We present the semantic notion of agentivity as a case study for probing such interactions. This suggests LMs may potentially serve as more useful tools for linguistic annotation, theory testing, and discovery.
  arXiv  Detail & Related papers  (2023-05-29T16:24:01Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• Subject Verb Agreement Error Patterns in Meaningless Sentences: Humans vs. BERT [64.40111510974957]
  We test whether meaning interferes with subject-verb number agreement in English. We generate semantically well-formed and nonsensical items. We find that BERT and humans are both sensitive to our semantic manipulation.
  arXiv  Detail & Related papers  (2022-09-21T17:57:23Z)
• Contextualized Semantic Distance between Highly Overlapped Texts [85.1541170468617]
  Overlapping frequently occurs in paired texts in natural language processing tasks like text editing and semantic similarity evaluation. This paper aims to address the issue with a mask-and-predict strategy. We take the words in the longest common sequence as neighboring words and use masked language modeling (MLM) to predict the distributions on their positions. Experiments on Semantic Textual Similarity show NDD to be more sensitive to various semantic differences, especially on highly overlapped paired texts.
  arXiv  Detail & Related papers  (2021-10-04T03:59:15Z)
• EDS-MEMBED: Multi-sense embeddings based on enhanced distributional semantic structures via a graph walk over word senses [0.0]
  We leverage the rich semantic structures in WordNet to enhance the quality of multi-sense embeddings. We derive new distributional semantic similarity measures for M-SE from prior ones. We report evaluation results on 11 benchmark datasets involving WSD and Word Similarity tasks.
  arXiv  Detail & Related papers  (2021-02-27T14:36:55Z)
• Exploring BERT's Sensitivity to Lexical Cues using Tests from Semantic Priming [8.08493736237816]
  We present a case study analyzing the pre-trained BERT model with tests informed by semantic priming. We find that BERT too shows "priming," predicting a word with greater probability when the context includes a related word versus an unrelated one. Follow-up analysis shows BERT to be increasingly distracted by related prime words as context becomes more informative.
  arXiv  Detail & Related papers  (2020-10-06T20:30:59Z)
• Pareto Probing: Trading Off Accuracy for Complexity [87.09294772742737]
  We argue for a probe metric that reflects the fundamental trade-off between probe complexity and performance. Our experiments with dependency parsing reveal a wide gap in syntactic knowledge between contextual and non-contextual representations.
  arXiv  Detail & Related papers  (2020-10-05T17:27:31Z)
• Syntactic Structure Distillation Pretraining For Bidirectional Encoders [49.483357228441434]
  We introduce a knowledge distillation strategy for injecting syntactic biases into BERT pretraining. We distill the approximate marginal distribution over words in context from the syntactic LM. Our findings demonstrate the benefits of syntactic biases, even in representation learners that exploit large amounts of data.
  arXiv  Detail & Related papers  (2020-05-27T16:44:01Z)
• Word Interdependence Exposes How LSTMs Compose Representations [18.34617849764921]
  Recent work in NLP shows that LSTM language models capture compositional structure in language data. We present a novel measure of interdependence between word meanings in an LSTM, based on their interactions at the internal gates.
  arXiv  Detail & Related papers  (2020-04-27T21:48:08Z)
• Quantifying the Contextualization of Word Representations with Semantic Class Probing [8.401007663676214]
  Pretrained language models have achieved a new state of the art on many NLP tasks, but there are still many open questions about how and why they work so well. We quantify the amount of contextualization, i.e., how well words are interpreted in context, by studying the extent to which semantic classes of a word can be inferred from its contextualized embeddings.
  arXiv  Detail & Related papers  (2020-04-25T17:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.