Understanding by Understanding Not: Modeling Negation in Language Models

• URL: http://arxiv.org/abs/2105.03519v1
• Date: Fri, 7 May 2021 21:58:35 GMT
• Title: Understanding by Understanding Not: Modeling Negation in Language Models
• Authors: Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni and Aaron Courville
• Abstract summary: Negation is a core construction in natural language. We propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences. We reduce the mean top1 error rate to 4% on the negated LAMA dataset.
• Score: 81.21351681735973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Negation is a core construction in natural language. Despite being very successful on many tasks, state-of-the-art pre-trained language models often handle negation incorrectly. To improve language models in this regard, we propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences from a raw text corpus. By training BERT with the resulting combined objective we reduce the mean top~1 error rate to 4% on the negated LAMA dataset. We also see some improvements on the negated NLI benchmarks.

Related papers

• Making Language Models Robust Against Negation [9.818585902859363]
  We propose a self-supervised method to make language models more robust against negation. We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks.
  arXiv  Detail & Related papers  (2025-02-11T17:18:47Z)
• Vision-Language Models Do Not Understand Negation [50.27667000027403]
  NegBench is a benchmark designed to evaluate negation understanding across 18 task variations and 79k examples spanning image, video, and medical datasets. We show that this approach can result in a 10% increase in recall on negated queries and a 40% boost in accuracy on multiple-choice questions with negated captions.
  arXiv  Detail & Related papers  (2025-01-16T09:55:42Z)
• This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models [4.017326849033009]
  We try to clarify the reasons for the sub-optimal performance of large language models understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability.
  arXiv  Detail & Related papers  (2023-10-24T15:38:21Z)
• Towards preserving word order importance through Forced Invalidation [80.33036864442182]
  We show that pre-trained language models are insensitive to word order. We propose Forced Invalidation to help preserve the importance of word order. Our experiments demonstrate that Forced Invalidation significantly improves the sensitivity of the models to word order.
  arXiv  Detail & Related papers  (2023-04-11T13:42:10Z)
• CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation [21.56001677478673]
  We present the first English reading comprehension dataset which requires reasoning about the implications of negated statements in paragraphs. CONDAQA features 14,182 question-answer pairs with over 200 unique negation cues. The best performing model on CONDAQA (UnifiedQA-v2-3b) achieves only 42% on our consistency metric, well below human performance which is 81%.
  arXiv  Detail & Related papers  (2022-11-01T06:10:26Z)
• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Quark: Controllable Text Generation with Reinforced Unlearning [68.07749519374089]
  Large-scale language models often learn behaviors that are misaligned with user expectations. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods.
  arXiv  Detail & Related papers  (2022-05-26T21:11:51Z)
• Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative Data [13.392212395386933]
  We find that within a model family, as the number of parameters, training epochs, and data set size increase, so does a model's ability to generalize to negative n-gram data. We propose a form of inductive bias that attenuates such undesirable signals with negative data distributions automatically learned from positive data.
  arXiv  Detail & Related papers  (2020-10-22T16:45:32Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models [32.183409062294466]
  We explore the utilities of explicit negative examples in training neural language models. We find that even with our direct learning signals the models still suffer from resolving agreement across an object-relative clause.
  arXiv  Detail & Related papers  (2020-04-06T07:47:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.