Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics

• URL: http://arxiv.org/abs/2404.03301v1
• Date: Thu, 4 Apr 2024 08:52:25 GMT
• Title: Probing Large Language Models for Scalar Adjective Lexical Semantics and Scalar Diversity Pragmatics
• Authors: Fangru Lin, Daniel Altshuler, Janet B. Pierrehumbert,
• Abstract summary: We probe different families of Large Language Models for their knowledge of the lexical semantics of scalar adjectives. We find that they encode rich lexical-semantic information about scalar adjectives. We also compare current models of different sizes and complexities and find that larger models are not always better.
• Score: 11.79778723980276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scalar adjectives pertain to various domain scales and vary in intensity within each scale (e.g. certain is more intense than likely on the likelihood scale). Scalar implicatures arise from the consideration of alternative statements which could have been made. They can be triggered by scalar adjectives and require listeners to reason pragmatically about them. Some scalar adjectives are more likely to trigger scalar implicatures than others. This phenomenon is referred to as scalar diversity. In this study, we probe different families of Large Language Models such as GPT-4 for their knowledge of the lexical semantics of scalar adjectives and one specific aspect of their pragmatics, namely scalar diversity. We find that they encode rich lexical-semantic information about scalar adjectives. However, the rich lexical-semantic knowledge does not entail a good understanding of scalar diversity. We also compare current models of different sizes and complexities and find that larger models are not always better. Finally, we explain our probing results by leveraging linguistic intuitions and model training objectives.

Related papers

• Are LLMs Models of Distributional Semantics? A Case Study on Quantifiers [14.797001158310092]
  We argue that distributional semantics models struggle with truth-conditional reasoning and symbolic processing. Contrary to expectations, we find that LLMs align more closely with human judgements on exact quantifiers versus vague ones.
  arXiv  Detail & Related papers  (2024-10-17T19:28:35Z)
• Not wacky vs. definitely wacky: A study of scalar adverbs in pretrained language models [0.0]
  Modern pretrained language models, such as BERT, RoBERTa and GPT-3 hold the promise of performing better on logical tasks than classic static word embeddings. We investigate the extent to which BERT, RoBERTa, GPT-2 and GPT-3 exhibit general, human-like, knowledge of these common words. We find that despite capturing some aspects of logical meaning, the models fall far short of human performance.
  arXiv  Detail & Related papers  (2023-05-25T18:56:26Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Language statistics at different spatial, temporal, and grammatical scales [48.7576911714538]
  We use data from Twitter to explore the rank diversity at different scales. The greatest changes come from variations in the grammatical scale. As the grammatical scale grows, the rank diversity curves vary more depending on the temporal and spatial scales.
  arXiv  Detail & Related papers  (2022-07-02T01:38:48Z)
• Scalar Adjective Identification and Multilingual Ranking [4.915907527975786]
  We introduce a new multilingual dataset in order to promote research on scalar adjectives in new languages. We perform a series of experiments and set performance baselines on this dataset, using monolingual and multilingual contextual language models. We introduce a new binary classification task for English scalar adjective identification.
  arXiv  Detail & Related papers  (2021-05-03T21:32:41Z)
• Probing Pretrained Language Models for Lexical Semantics [76.73599166020307]
  We present a systematic empirical analysis across six typologically diverse languages and five different lexical tasks. Our results indicate patterns and best practices that hold universally, but also point to prominent variations across languages and tasks.
  arXiv  Detail & Related papers  (2020-10-12T14:24:01Z)
• Do Language Embeddings Capture Scales? [54.1633257459927]
  We show that pretrained language models capture a significant amount of information about the scalar magnitudes of objects. We identify contextual information in pre-training and numeracy as two key factors affecting their performance.
  arXiv  Detail & Related papers  (2020-10-11T21:11:09Z)
• Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model [66.84264870118723]
  We present the first purely corpus-driven model of multi-lingual adjective ordering in the form of a latent-variable model. We provide strong converging evidence for the existence of universal, cross-linguistic, hierarchical adjective ordering tendencies.
  arXiv  Detail & Related papers  (2020-10-09T18:27:55Z)
• BERT Knows Punta Cana is not just beautiful, it's gorgeous: Ranking Scalar Adjectives with Contextualised Representations [6.167728295758172]
  We propose a novel BERT-based approach to intensity detection for scalar adjectives. We model intensity by vectors directly derived from contextualised representations and show they can successfully rank scalar adjectives.
  arXiv  Detail & Related papers  (2020-10-06T13:05:47Z)
• Are All Good Word Vector Spaces Isomorphic? [79.04509759167952]
  We show that variance in performance across language pairs is not only due to typological differences, but can mostly be attributed to the size of the monolingual resources available.
  arXiv  Detail & Related papers  (2020-04-08T15:49:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.