Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding

• URL: http://arxiv.org/abs/2603.29258v1
• Date: Tue, 31 Mar 2026 04:48:52 GMT
• Title: Omni-NegCLIP: Enhancing CLIP with Front-Layer Contrastive Fine-Tuning for Comprehensive Negation Understanding
• Abstract summary: Recent studies have shown that vision-Language Models (VLMs) perform poorly in understanding negation expressions.<n>We propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation.<n> Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have demonstrated strong capabilities across a wide range of multimodal tasks. However, recent studies have shown that VLMs, such as CLIP, perform poorly in understanding negation expressions, which are common in natural language. In this work, we propose Omni-NegCLIP, a fine-tuned CLIP model that improves CLIP's understanding of two types of negation, namely presence-based negation and absence-based negation, which correspond to negated expressions of objects that are actually present in an image and those that may plausibly exist in an image but are in fact absent, respectively, by modifying CLIP's original InfoNCE contrastive loss. Specifically, we design a presence-based contrastive objective that pulls image embeddings closer to their original caption embeddings while pushing them away from the corresponding presence-based negated caption embeddings, and an absence-based contrastive objective that aligns image embeddings with both original and absence-based negated caption embeddings while maintaining a semantic distinction between the two text embeddings. Based on our observation that the front transformer layers of CLIP text encoder have stronger learning ability for negated text than the later layers, we fine-tune the front transformer layers of the CLIP text encoder at each training step using the combined contrastive objective. Experimental results show that, compared with pretrained CLIP, Omni-NegCLIP improves performance on presence-based negation and absence-based negation tasks by up to 52.65% and 12.50%, respectively, without sacrificing general capability in image-text retrieval and even improving it by up to 19.62%. Compared with prior works, Omni-NegCLIP demonstrates a more comprehensive ability to understand multiple types of negation tasks.

Related papers

• Finetune Like You Pretrain: Boosting Zero-shot Adversarial Robustness in Vision-language Models [89.0460992131069]
  We propose AdvFLYP, which follows the training recipe of CLIP's pretraining process when performing adversarial finetuning to the model.<n>Specifically, AdvFLYP finetunes CLIP with adversarial images created based on image-text pairs collected from the web, and match them with their corresponding texts via a contrastive loss.<n>We show that logit- and feature-level regularisation terms benefit robustness and clean accuracy, respectively.
  arXiv  Detail & Related papers  (2026-04-13T14:54:25Z)
• Contrastive vision-language learning with paraphrasing and negation [0.018665975431697426]
  Contrastive language-Image Pre-training (CLIP) trains two neural networks in contrastive manner to align their image and text embeddings in a shared latent space.<n>Results evaluating CLIP on negated or paraphrased text have shown mixed performance because negation changes meaning radically with minimal lexical changes, while paraphrasing can create very different textual expressions with the same intended meaning.<n>This paper proposes a new CLIP contrastive loss function accounting for both paraphrasing and negation, and applies LLM-generated training triples consisting of original, paraphrased and negated textual captions to CLIP-like training models.
  arXiv  Detail & Related papers  (2025-11-20T16:41:36Z)
• VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction Editing Data and Long Captions [16.90061119174727]
  We present CLIP-IN, a novel framework that bolsters CLIP's fine-grained perception through two core innovations.<n> Firstly, we leverage instruction-editing datasets, originally designed for image manipulation, as a unique source of hard negative image-text pairs.<n> Secondly, CLIP-IN incorporates long captions, utilizing rotary positional encodings to capture rich semantic context often missed by standard CLIP.
  arXiv  Detail & Related papers  (2025-08-04T11:57:10Z)
• TNG-CLIP:Training-Time Negation Data Generation for Negation Awareness of CLIP [27.33315565500182]
  We introduce a training-time negation data generation pipeline for vision-language models (VLMs)<n>We propose the first benchmark, Neg-TtoI, for evaluating text-to-image generation models on prompts containing negation.<n>We show that our proposed method, TNG-CLIP, achieves SOTA performance on diverse negation benchmarks of image-to-text matching, text-to-image retrieval, and image generation.
  arXiv  Detail & Related papers  (2025-05-24T00:02:48Z)
• Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP [57.33324843049638]
  We introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions.<n>Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality.<n>Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately.
  arXiv  Detail & Related papers  (2025-01-19T01:17:05Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Finetuning CLIP to Reason about Pairwise Differences [52.028073305958074]
  We propose an approach to train vision-language models such as CLIP in a contrastive manner to reason about differences in embedding space.<n>We finetune CLIP so that text descriptions of differences between images correspond to their difference in image embedding space.<n>Our approach yields significantly improved capabilities in ranking images by a certain attribute, and improved zeroshot classification performance on many downstream image classification tasks.
  arXiv  Detail & Related papers  (2024-09-15T13:02:14Z)
• Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations [43.484570564890866]
  Existing vision-language models (VLMs) treat text descriptions as a unit, confusing individual concepts in a prompt. We present CC-Neg, a dataset containing 228,246 images, true captions and their corresponding negated captions. Using CC-Neg along with modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework, has an improved understanding of negations.
  arXiv  Detail & Related papers  (2024-03-29T17:33:42Z)
• Non-Contrastive Learning Meets Language-Image Pre-Training [145.6671909437841]
  We study the validity of non-contrastive language-image pre-training (nCLIP) We introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics.
  arXiv  Detail & Related papers  (2022-10-17T17:57:46Z)
• No Token Left Behind: Explainability-Aided Image Classification and Generation [79.4957965474334]
  We present a novel explainability-based approach, which adds a loss term to ensure that CLIP focuses on all relevant semantic parts of the input. Our method yields an improvement in the recognition rate, without additional training or fine-tuning.
  arXiv  Detail & Related papers  (2022-04-11T07:16:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.