Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP

• URL: http://arxiv.org/abs/2501.10913v3
• Date: Wed, 27 Aug 2025 05:29:07 GMT
• Title: Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP
• Abstract summary: We introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions.<n>Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality.<n>Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately.
• Score: 57.33324843049638
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While CLIP has significantly advanced multimodal understanding by bridging vision and language, the inability to grasp negation - such as failing to differentiate concepts like "parking" from "no parking" - poses substantial challenges. By analyzing the data used in the public CLIP model's pre-training, we posit this limitation stems from a lack of negation-inclusive data. To address this, we introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions. Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality. Moreover, to enable a comprehensive evaluation of negation understanding, we propose NegRefCOCOg-a benchmark tailored to test VLMs' ability to interpret negation across diverse expressions and positions within a sentence. Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately. Additionally, NegationCLIP's enhanced negation awareness has practical applications across various multimodal tasks, demonstrated by performance gains in text-to-image generation and referring image segmentation.

Related papers

• HANCLIP: A Family of Hyperbolic Angular Negation Vision Language Models [3.4657033095341845]
  This work introduces HANCLIP (Hyperbolic + Angular + Negation), a family of vision-language models that explicitly restructures the embedding space to encode "what an image is not" alongside "what it is"<n>HANCLIP is trained on a compact set of 20,000 image-text quadruplets and combines a hyperbolic formulation, which models hierarchical semantic relations and asymmetries, with an angular triplet objective that drives systematic separation between negated descriptions and their corresponding positives.<n>Experiments show that HANCLIP delivers consistent gains on the negation-focused NegBench benchmark, while maintaining competitive or improved performance on standard classification
  arXiv  Detail & Related papers  (2026-06-22T18:25:37Z)
• Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding [0.0]
  Recent studies have shown that vision-Language Models (VLMs) perform poorly in understanding negation expressions.<n>We propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation.<n> Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively.
  arXiv  Detail & Related papers  (2026-03-31T04:48:52Z)
• When Negation Is a Geometry Problem in Vision-Language Models [32.51815690470519]
  Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries.<n>We investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which excel at understanding simple yes/no questions about image content.
  arXiv  Detail & Related papers  (2026-03-20T23:06:23Z)
• Negation-Aware Test-Time Adaptation for Vision-Language Models [26.043679706381646]
  We study a practical but less-touched problem in Vision-Language Models (VLMs)<n>Many real-world applications require models to explicitly identify what is false or non-existent.<n>We propose a Negation-Aware Test-Time Adaptation (NEAT) method to efficiently adjust distribution-related parameters during inference.
  arXiv  Detail & Related papers  (2025-07-25T08:25:48Z)
• Learning Robust Negation Text Representations [60.23044940174016]
  We propose a strategy to improve negation of text encoders using diverse patterns of negation and hedging.<n>We observe large improvement in negation understanding capabilities while maintaining competitive performance on general benchmarks.<n>Our method can be adapted to LLMs, leading to improved performance on negation benchmarks.
  arXiv  Detail & Related papers  (2025-07-17T04:48:54Z)
• Vision-Language Models Do Not Understand Negation [50.27667000027403]
  NegBench is a benchmark designed to evaluate negation understanding across 18 task variations and 79k examples spanning image, video, and medical datasets.<n>We show that this approach can result in a 10% increase in recall on negated queries and a 40% boost in accuracy on multiple-choice questions with negated captions.
  arXiv  Detail & Related papers  (2025-01-16T09:55:42Z)
• Revisiting subword tokenization: A case study on affixal negation in large language models [57.75279238091522]
  We measure the impact of affixal negation on modern English large language models (LLMs) We conduct experiments using LLMs with different subword tokenization methods. We show that models can, on the whole, reliably recognize the meaning of affixal negation.
  arXiv  Detail & Related papers  (2024-04-03T03:14:27Z)
• Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations [43.484570564890866]
  Existing vision-language models (VLMs) treat text descriptions as a unit, confusing individual concepts in a prompt. We present CC-Neg, a dataset containing 228,246 images, true captions and their corresponding negated captions. Using CC-Neg along with modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework, has an improved understanding of negations.
  arXiv  Detail & Related papers  (2024-03-29T17:33:42Z)
• This is not a Dataset: A Large Negation Benchmark to Challenge Large Language Models [4.017326849033009]
  We try to clarify the reasons for the sub-optimal performance of large language models understanding negation. We introduce a large semi-automatically generated dataset of circa 400,000 descriptive sentences about commonsense knowledge. We have used our dataset with the largest available open LLMs in a zero-shot approach to grasp their generalization and inference capability.
  arXiv  Detail & Related papers  (2023-10-24T15:38:21Z)
• Language models are not naysayers: An analysis of language models on negation benchmarks [58.32362243122714]
  We evaluate the ability of current-generation auto-regressive language models to handle negation. We show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.
  arXiv  Detail & Related papers  (2023-06-14T01:16:37Z)
• Positive-Negative Equal Contrastive Loss for Semantic Segmentation [8.664491798389662]
  Previous works commonly design plug-and-play modules and structural losses to effectively extract and aggregate the global context. We propose Positive-Negative Equal contrastive loss (PNE loss), which increases the latent impact of positive embedding on the anchor and treats the positive as well as negative sample pairs equally. We conduct comprehensive experiments and achieve state-of-the-art performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-04T13:51:29Z)
• Improving negation detection with negation-focused pre-training [58.32362243122714]
  Negation is a common linguistic feature that is crucial in many language understanding tasks. Recent work has shown that state-of-the-art NLP models underperform on samples containing negation. We propose a new negation-focused pre-training strategy, involving targeted data augmentation and negation masking.
  arXiv  Detail & Related papers  (2022-05-09T02:41:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.