Pragmatic inference of scalar implicature by LLMs

• URL: http://arxiv.org/abs/2408.06673v1
• Date: Tue, 13 Aug 2024 06:52:29 GMT
• Title: Pragmatic inference of scalar implicature by LLMs
• Abstract summary: This study investigates how Large Language Models (LLMs) engage in pragmatic inference of implicature, such as some. BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model. GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study investigates how Large Language Models (LLMs), particularly BERT (Devlin et al., 2019) and GPT-2 (Radford et al., 2019), engage in pragmatic inference of scalar implicature, such as some. Two sets of experiments were conducted using cosine similarity and next sentence/token prediction as experimental methods. The results in experiment 1 showed that, both models interpret some as pragmatic implicature not all in the absence of context, aligning with human language processing. In experiment 2, in which Question Under Discussion (QUD) was presented as a contextual cue, BERT showed consistent performance regardless of types of QUDs, while GPT-2 encountered processing difficulties since a certain type of QUD required pragmatic inference for implicature. The findings revealed that, in terms of theoretical approaches, BERT inherently incorporates pragmatic implicature not all within the term some, adhering to Default model (Levinson, 2000). In contrast, GPT-2 seems to encounter processing difficulties in inferring pragmatic implicature within context, consistent with Context-driven model (Sperber and Wilson, 2002).

Related papers

• Evaluating Pragmatic Reasoning in Large Language Models: Evidence from Scalar Diversity [2.538209532048867]
  This study compares direct probability measurement and metalinguistic prompting across multiple models and experimental settings.<n>The results show that neither evaluation method consistently outperforms the other and that pragmatic behavior varies substantially across model families, prompting strategies, and task structures.
  arXiv  Detail & Related papers  (2026-05-09T16:28:50Z)
• Do Language Models Know Theo Has a Wife? Investigating the Proviso Problem [3.219880761967806]
  We introduce a diagnostic dataset designed to probe presupposition projection in conditionals.<n>Results show that models broadly align with human judgments but rely on shallow pattern matching rather than semantic or pragmatic reasoning.
  arXiv  Detail & Related papers  (2026-03-09T13:21:56Z)
• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• Understanding In-Context Learning with a Pelican Soup Framework [27.144616560712493]
  We propose a theoretical framework to explain in-context learning for natural language processing. Our results demonstrate the efficacy of our framework to explain in-context learning.
  arXiv  Detail & Related papers  (2024-02-16T03:20:14Z)
• DiPlomat: A Dialogue Dataset for Situated Pragmatic Reasoning [89.92601337474954]
  Pragmatic reasoning plays a pivotal role in deciphering implicit meanings that frequently arise in real-life conversations. We introduce a novel challenge, DiPlomat, aiming at benchmarking machines' capabilities on pragmatic reasoning and situated conversational understanding.
  arXiv  Detail & Related papers  (2023-06-15T10:41:23Z)
• Prototyping Theories with ChatGPT: Experiment with the Technology Acceptance Model [0.0]
  The objective of the studies is to assess ChatGPT's ability to comprehend theoretical concepts and differentiate between constructs. The results of the experiments indicate that ChatGPT can generate responses aligned with the constructs of the Acceptance Technology Model (TAM) The experiments reveal potential biases in the generated samples, particularly regarding gender and usage experiences.
  arXiv  Detail & Related papers  (2023-06-04T23:55:53Z)
• How Does In-Context Learning Help Prompt Tuning? [55.78535874154915]
  Fine-tuning large language models is becoming ever more impractical due to their rapidly-growing scale. This motivates the use of parameter-efficient adaptation methods such as prompt tuning (PT), which adds a small number of tunable embeddings to an otherwise frozen model. Recently, Singhal et al. (2022) propose instruction prompt tuning'' (IPT), which combines PT with ICL by concatenating a natural language demonstration with learned prompt embeddings.
  arXiv  Detail & Related papers  (2023-02-22T17:45:12Z)
• Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement [4.995343972237368]
  The long-distance agreement is increasingly used to assess the syntactic generalization of Neural Language Models. Much work has shown that transformers are capable of high accuracy in varied agreement tasks. This work contrasts how they handle two superficially similar but theoretically distinct agreement phenomena.
  arXiv  Detail & Related papers  (2022-12-08T19:10:46Z)
• Did the Cat Drink the Coffee? Challenging Transformers with Generalized Event Knowledge [59.22170796793179]
  Transformers Language Models (TLMs) were tested on a benchmark for the textitdynamic estimation of thematic fit Our results show that TLMs can reach performances that are comparable to those achieved by SDM. However, additional analysis consistently suggests that TLMs do not capture important aspects of event knowledge.
  arXiv  Detail & Related papers  (2021-07-22T20:52:26Z)
• Infusing Finetuning with Semantic Dependencies [62.37697048781823]
  We show that, unlike syntax, semantics is not brought to the surface by today's pretrained models. We then use convolutional graph encoders to explicitly incorporate semantic parses into task-specific finetuning.
  arXiv  Detail & Related papers  (2020-12-10T01:27:24Z)
• A Matter of Framing: The Impact of Linguistic Formalism on Probing Results [69.36678873492373]
  Deep pre-trained contextualized encoders like BERT (Delvin et al.) demonstrate remarkable performance on a range of downstream tasks. Recent research in probing investigates the linguistic knowledge implicitly learned by these models during pre-training. Can the choice of formalism affect probing results? We find linguistically meaningful differences in the encoding of semantic role- and proto-role information by BERT depending on the formalism.
  arXiv  Detail & Related papers  (2020-04-30T17:45:16Z)
• Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition [17.642255516887968]
  Natural language inference (NLI) is an increasingly important task for natural language understanding. The ability of NLI models to make pragmatic inferences remains understudied. We evaluate whether BERT, InferSent, and BOW NLI models trained on MultiNLI learn to make pragmatic inferences.
  arXiv  Detail & Related papers  (2020-04-07T01:20:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.