Can large language models generate salient negative statements?
- URL: http://arxiv.org/abs/2305.16755v2
- Date: Thu, 21 Sep 2023 13:36:03 GMT
- Title: Can large language models generate salient negative statements?
- Authors: Hiba Arnaout, Simon Razniewski
- Abstract summary: We examine the ability of large language models to generate salient (interesting) negative statements about real-world entities.
We probe the LLMs using zero- and k-shot unconstrained probes, and compare with traditional methods for negation generation.
We measure the correctness and salience of the generated lists about subjects from different domains.
- Score: 18.577880767789097
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We examine the ability of large language models (LLMs) to generate salient
(interesting) negative statements about real-world entities; an emerging
research topic of the last few years. We probe the LLMs using zero- and k-shot
unconstrained probes, and compare with traditional methods for negation
generation, i.e., pattern-based textual extractions and knowledge-graph-based
inferences, as well as crowdsourced gold statements. We measure the correctness
and salience of the generated lists about subjects from different domains. Our
evaluation shows that guided probes do in fact improve the quality of generated
negatives, compared to the zero-shot variant. Nevertheless, using both prompts,
LLMs still struggle with the notion of factuality of negatives, frequently
generating many ambiguous statements, or statements with negative keywords but
a positive meaning.
Related papers
- Generating Diverse Negations from Affirmative Sentences [0.999726509256195]
Negations are important in real-world applications as they encode negative polarity in verb phrases, clauses, or other expressions.
We propose NegVerse, a method that tackles the lack of negation datasets by producing a diverse range of negation types.
We provide new rules for masking parts of sentences where negations are most likely to occur, based on syntactic structure.
We also propose a filtering mechanism to identify negation cues and remove degenerate examples, producing a diverse range of meaningful perturbations.
arXiv Detail & Related papers (2024-10-30T21:25:02Z) - Large Language Models are Skeptics: False Negative Problem of Input-conflicting Hallucination [36.01680298955394]
We identify a new category of bias that induces input-conflicting hallucinations.
We show that large language models (LLMs) generate responses inconsistent with the content of the input context.
arXiv Detail & Related papers (2024-06-20T01:53:25Z) - Revisiting subword tokenization: A case study on affixal negation in large language models [57.75279238091522]
We measure the impact of affixal negation on modern English large language models (LLMs)
We conduct experiments using LLMs with different subword tokenization methods.
We show that models can, on the whole, reliably recognize the meaning of affixal negation.
arXiv Detail & Related papers (2024-04-03T03:14:27Z) - Generating Enhanced Negatives for Training Language-Based Object Detectors [86.1914216335631]
We propose to leverage the vast knowledge built into modern generative models to automatically build negatives that are more relevant to the original data.
Specifically, we use large-language-models to generate negative text descriptions, and text-to-image diffusion models to also generate corresponding negative images.
Our experimental analysis confirms the relevance of the generated negative data, and its use in language-based detectors improves performance on two complex benchmarks.
arXiv Detail & Related papers (2023-12-29T23:04:00Z) - This is not a Dataset: A Large Negation Benchmark to Challenge Large
Language Models [4.017326849033009]
We try to clarify the reasons for the sub-optimal performance of large language models understanding negation.
We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge.
We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability.
arXiv Detail & Related papers (2023-10-24T15:38:21Z) - Language models are not naysayers: An analysis of language models on
negation benchmarks [58.32362243122714]
We evaluate the ability of current-generation auto-regressive language models to handle negation.
We show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.
arXiv Detail & Related papers (2023-06-14T01:16:37Z) - Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal
Negation [59.307534363825816]
Negation is poorly captured by current language models, although the extent of this problem is not widely understood.
We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods.
arXiv Detail & Related papers (2022-10-06T23:39:01Z) - Improving negation detection with negation-focused pre-training [58.32362243122714]
Negation is a common linguistic feature that is crucial in many language understanding tasks.
Recent work has shown that state-of-the-art NLP models underperform on samples containing negation.
We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking.
arXiv Detail & Related papers (2022-05-09T02:41:11Z) - Understanding by Understanding Not: Modeling Negation in Language Models [81.21351681735973]
Negation is a core construction in natural language.
We propose to augment the language modeling objective with an unlikelihood objective that is based on negated generic sentences.
We reduce the mean top1 error rate to 4% on the negated LAMA dataset.
arXiv Detail & Related papers (2021-05-07T21:58:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.