Bidirectional Transformer Representations of (Spanish) Ambiguous Words in Context: A New Lexical Resource and Empirical Analysis

• URL: http://arxiv.org/abs/2406.14678v1
• Date: Thu, 20 Jun 2024 18:58:11 GMT
• Title: Bidirectional Transformer Representations of (Spanish) Ambiguous Words in Context: A New Lexical Resource and Empirical Analysis
• Authors: Pamela D. Rivière, Anne L. Beatty-Martínez, Sean Trott,
• Abstract summary: Few studies have compared large language models' contextualized word embeddings for languages beyond English. We evaluate multiple bidirectional transformers' (BERTs') semantic representations of Spanish ambiguous nouns in context. We find that various BERT-based LLMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark.
• Score: 2.2530496464901106
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Lexical ambiguity -- where a single wordform takes on distinct, context-dependent meanings -- serves as a useful tool to compare across different large language models' (LLMs') ability to form distinct, contextualized representations of the same stimulus. Few studies have systematically compared LLMs' contextualized word embeddings for languages beyond English. Here, we evaluate multiple bidirectional transformers' (BERTs') semantic representations of Spanish ambiguous nouns in context. We develop a novel dataset of minimal-pair sentences evoking the same or different sense for a target ambiguous noun. In a pre-registered study, we collect contextualized human relatedness judgments for each sentence pair. We find that various BERT-based LLMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark, and for Spanish -- unlike English -- model scale is uncorrelated with performance. We also identify stereotyped trajectories of target noun disambiguation as a proportion of traversal through a given LLM family's architecture, which we partially replicate in English. We contribute (1) a dataset of controlled, Spanish sentence stimuli with human relatedness norms, and (2) to our evolving understanding of the impact that LLM specification (architectures, training protocols) exerts on contextualized embeddings.

Related papers

• MINERS: Multilingual Language Models as Semantic Retrievers [23.686762008696547]
  This paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual language models in semantic retrieval tasks. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-06-11T16:26:18Z)
• How well do distributed representations convey contextual lexical semantics: a Thesis Proposal [3.3585951129432323]
  In this thesis, we examine the efficacy of distributed representations from modern neural networks in encoding lexical meaning. We identify four sources of ambiguity based on the relatedness and similarity of meanings influenced by context. We then aim to evaluate these sources by collecting or constructing multilingual datasets, leveraging various language models, and employing linguistic analysis tools.
  arXiv  Detail & Related papers  (2024-06-02T14:08:51Z)
• We're Afraid Language Models Aren't Modeling Ambiguity [136.8068419824318]
  Managing ambiguity is a key part of human language understanding. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence. We show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity.
  arXiv  Detail & Related papers  (2023-04-27T17:57:58Z)
• Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation [70.58243648754507]
  We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR) Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic similarity and transfer tasks.
  arXiv  Detail & Related papers  (2022-10-18T11:37:36Z)
• Transition-based Abstract Meaning Representation Parsing with Contextual Embeddings [0.0]
  We study a way of combing two of the most successful routes to meaning of language--statistical language models and symbolic semantics formalisms--in the task of semantic parsing. We explore the utility of incorporating pretrained context-aware word embeddings--such as BERT and RoBERTa--in the problem of parsing.
  arXiv  Detail & Related papers  (2022-06-13T15:05:24Z)
• Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution [124.99894592871385]
  We present a large-scale comparative study of lexical substitution methods employing both old and most recent language models. We show that already competitive results achieved by SOTA LMs/MLMs can be further substantially improved if information about the target word is injected properly.
  arXiv  Detail & Related papers  (2022-06-07T16:16:19Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• Exploring the Representation of Word Meanings in Context: A Case Study on Homonymy and Synonymy [0.0]
  We assess the ability of both static and contextualized models to adequately represent different lexical-semantic relations. Experiments are performed in Galician, Portuguese, English, and Spanish.
  arXiv  Detail & Related papers  (2021-06-25T10:54:23Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• BURT: BERT-inspired Universal Representation from Learning Meaningful Segment [46.51685959045527]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present a universal representation model, BURT, to encode different levels of linguistic unit into the same vector space. Specifically, we extract and mask meaningful segments based on point-wise mutual information (PMI) to incorporate different granular objectives into the pre-training stage.
  arXiv  Detail & Related papers  (2020-12-28T16:02:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.