Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?

• URL: http://arxiv.org/abs/2405.07099v1
• Date: Sat, 11 May 2024 21:50:56 GMT
• Title: Do Pretrained Contextual Language Models Distinguish between Hebrew Homograph Analyses?
• Authors: Avi Shmidman, Cheyn Shmuel Shmidman, Dan Bareket, Moshe Koppel, Reut Tsarfaty,
• Abstract summary: We study the extent to which Hebrew homographs can be disambiguated and analyzed using pre-trained language models. We show that contemporary Hebrew contextualized embeddings outperform non-contextualized embeddings. We also show that these embeddings are equally effective for homographs of both balanced and skewed distributions.
• Score: 12.631897904322676
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Semitic morphologically-rich languages (MRLs) are characterized by extreme word ambiguity. Because most vowels are omitted in standard texts, many of the words are homographs with multiple possible analyses, each with a different pronunciation and different morphosyntactic properties. This ambiguity goes beyond word-sense disambiguation (WSD), and may include token segmentation into multiple word units. Previous research on MRLs claimed that standardly trained pre-trained language models (PLMs) based on word-pieces may not sufficiently capture the internal structure of such tokens in order to distinguish between these analyses. Taking Hebrew as a case study, we investigate the extent to which Hebrew homographs can be disambiguated and analyzed using PLMs. We evaluate all existing models for contextualized Hebrew embeddings on a novel Hebrew homograph challenge sets that we deliver. Our empirical results demonstrate that contemporary Hebrew contextualized embeddings outperform non-contextualized embeddings; and that they are most effective for disambiguating segmentation and morphosyntactic features, less so regarding pure word-sense disambiguation. We show that these embeddings are more effective when the number of word-piece splits is limited, and they are more effective for 2-way and 3-way ambiguities than for 4-way ambiguity. We show that the embeddings are equally effective for homographs of both balanced and skewed distributions, whether calculated as masked or unmasked tokens. Finally, we show that these embeddings are as effective for homograph disambiguation with extensive supervised training as with a few-shot setup.

Related papers

• Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis [2.2530496464901106]
  We evaluate semantic representations of Spanish ambiguous nouns in context in a suite of Spanish-language monolingual and multilingual BERT-based models. We find that various BERT-based LMs' contextualized semantic representations capture some variance in human judgments but fall short of the human benchmark.
  arXiv  Detail & Related papers  (2024-06-20T18:58:11Z)
• Persian Homograph Disambiguation: Leveraging ParsBERT for Enhanced Sentence Understanding with a Novel Word Disambiguation Dataset [0.0]
  We introduce a novel dataset tailored for Persian homograph disambiguation. Our work encompasses a thorough exploration of various embeddings, evaluated through the cosine similarity method. We scrutinize the models' performance in terms of Accuracy, Recall, and F1 Score.
  arXiv  Detail & Related papers  (2024-05-24T14:56:36Z)
• Breaking Down Word Semantics from Pre-trained Language Models through Layer-wise Dimension Selection [0.0]
  This paper aims to disentangle semantic sense from BERT by applying a binary mask to middle outputs across the layers. The disentangled embeddings are evaluated through binary classification to determine if the target word in two different sentences has the same meaning.
  arXiv  Detail & Related papers  (2023-10-08T11:07:19Z)
• We're Afraid Language Models Aren't Modeling Ambiguity [136.8068419824318]
  Managing ambiguity is a key part of human language understanding. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence. We show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity.
  arXiv  Detail & Related papers  (2023-04-27T17:57:58Z)
• A Latent-Variable Model for Intrinsic Probing [93.62808331764072]
  We propose a novel latent-variable formulation for constructing intrinsic probes. We find empirical evidence that pre-trained representations develop a cross-lingually entangled notion of morphosyntax.
  arXiv  Detail & Related papers  (2022-01-20T15:01:12Z)
• More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models [71.42030830910227]
  We propose a new metric for measuring the clustering quality in settings where the models differ. We show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.
  arXiv  Detail & Related papers  (2021-08-24T14:08:19Z)
• A Novel Challenge Set for Hebrew Morphological Disambiguation and Diacritics Restoration [8.704581499692651]
  We offer a challenge set for Hebrew homographs -- the first of its kind. We show that the current SOTA of Hebrew disambiguation performs poorly on cases of unbalanced ambiguity. We achieve a new state-of-the-art for all 21 words, improving the overall average F1 score from 0.67 to 0.95.
  arXiv  Detail & Related papers  (2020-10-06T16:34:03Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Speakers Fill Lexical Semantic Gaps with Context [65.08205006886591]
  We operationalise the lexical ambiguity of a word as the entropy of meanings it can take. We find significant correlations between our estimate of ambiguity and the number of synonyms a word has in WordNet. This suggests that, in the presence of ambiguity, speakers compensate by making contexts more informative.
  arXiv  Detail & Related papers  (2020-10-05T17:19:10Z)
• On the Language Neutrality of Pre-trained Multilingual Representations [70.93503607755055]
  We investigate the language-neutrality of multilingual contextual embeddings directly and with respect to lexical semantics. Our results show that contextual embeddings are more language-neutral and, in general, more informative than aligned static word-type embeddings. We show how to reach state-of-the-art accuracy on language identification and match the performance of statistical methods for word alignment of parallel sentences.
  arXiv  Detail & Related papers  (2020-04-09T19:50:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.