Probing structural constraints of negation in Pretrained Language Models

• URL: http://arxiv.org/abs/2408.03070v1
• Date: Tue, 6 Aug 2024 09:54:49 GMT
• Title: Probing structural constraints of negation in Pretrained Language Models
• Authors: David Kletz, Marie Candito, Pascal Amsili,
• Abstract summary: We use probes to identify which contextual representations best encode the presence of negation in a sentence. We find that contextual representations of tokens inside the negation scope do allow for (i) a better prediction of the presence of not compared to those outside the scope. Yet, further control experiments reveal that the presence of other lexical items is also better captured when using the contextual representation of a token within the same syntactic clause.
• Score: 1.8749305679160366
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contradictory results about the encoding of the semantic impact of negation in pretrained language models (PLMs). have been drawn recently (e.g. Kassner and Sch{\"u}tze (2020); Gubelmann and Handschuh (2022)). In this paper we focus rather on the way PLMs encode negation and its formal impact, through the phenomenon of the Negative Polarity Item (NPI) licensing in English. More precisely, we use probes to identify which contextual representations best encode 1) the presence of negation in a sentence, and 2) the polarity of a neighboring masked polarity item. We find that contextual representations of tokens inside the negation scope do allow for (i) a better prediction of the presence of not compared to those outside the scope and (ii) a better prediction of the right polarity of a masked polarity item licensed by not, although the magnitude of the difference varies from PLM to PLM. Importantly, in both cases the trend holds even when controlling for distance to not. This tends to indicate that the embeddings of these models do reflect the notion of negation scope, and do encode the impact of negation on NPI licensing. Yet, further control experiments reveal that the presence of other lexical items is also better captured when using the contextual representation of a token within the same syntactic clause than outside from it, suggesting that PLMs simply capture the more general notion of syntactic clause.

Related papers

• Generating Diverse Negations from Affirmative Sentences [0.999726509256195]
  Negations are important in real-world applications as they encode negative polarity in verb phrases, clauses, or other expressions. We propose NegVerse, a method that tackles the lack of negation datasets by producing a diverse range of negation types. We provide new rules for masking parts of sentences where negations are most likely to occur, based on syntactic structure. We also propose a filtering mechanism to identify negation cues and remove degenerate examples, producing a diverse range of meaningful perturbations.
  arXiv  Detail & Related papers  (2024-10-30T21:25:02Z)
• The Self-Contained Negation Test Set [1.8749305679160366]
  We build on Gubelmann and Handschuh (2022), which studies the modification of PLMs' predictions as a function of the polarity of inputs, in English. This test uses self-contained'' inputs ending with a masked position. We propose an improved version, the Self-Contained Neg Test, which is more controlled, more systematic, and entirely based on examples forming minimal pairs.
  arXiv  Detail & Related papers  (2024-08-21T09:38:15Z)
• Revisiting subword tokenization: A case study on affixal negation in large language models [57.75279238091522]
  We measure the impact of affixal negation on modern English large language models (LLMs) We conduct experiments using LLMs with different subword tokenization methods. We show that models can, on the whole, reliably recognize the meaning of affixal negation.
  arXiv  Detail & Related papers  (2024-04-03T03:14:27Z)
• Language models are not naysayers: An analysis of language models on negation benchmarks [58.32362243122714]
  We evaluate the ability of current-generation auto-regressive language models to handle negation. We show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.
  arXiv  Detail & Related papers  (2023-06-14T01:16:37Z)
• Can large language models generate salient negative statements? [18.577880767789097]
  We examine the ability of large language models to generate salient (interesting) negative statements about real-world entities. We probe the LLMs using zero- and k-shot unconstrained probes, and compare with traditional methods for negation generation. We measure the correctness and salience of the generated lists about subjects from different domains.
  arXiv  Detail & Related papers  (2023-05-26T09:13:59Z)
• Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation [59.307534363825816]
  Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods.
  arXiv  Detail & Related papers  (2022-10-06T23:39:01Z)
• Improving negation detection with negation-focused pre-training [58.32362243122714]
  Negation is a common linguistic feature that is crucial in many language understanding tasks. Recent work has shown that state-of-the-art NLP models underperform on samples containing negation. We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking.
  arXiv  Detail & Related papers  (2022-05-09T02:41:11Z)
• Debiased Contrastive Learning of Unsupervised Sentence Representations [88.58117410398759]
  Contrastive learning is effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. Previous works mostly adopt in-batch negatives or sample from training data at random. We present a new framework textbfDCLR to alleviate the influence of these improper negatives.
  arXiv  Detail & Related papers  (2022-05-02T05:07:43Z)
• Understanding by Understanding Not: Modeling Negation in Language Models [81.21351681735973]
  Negation is a core construction in natural language. We propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences. We reduce the mean top1 error rate to 4% on the negated LAMA dataset.
  arXiv  Detail & Related papers  (2021-05-07T21:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.