When Negation Is a Geometry Problem in Vision-Language Models

• URL: http://arxiv.org/abs/2603.20554v1
• Date: Fri, 20 Mar 2026 23:06:23 GMT
• Title: When Negation Is a Geometry Problem in Vision-Language Models
• Abstract summary: Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries.<n>We investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which excel at understanding simple yes/no questions about image content.
• Score: 32.51815690470519
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Joint Vision-Language Embedding models such as CLIP typically fail at understanding negation in text queries - for example, failing to distinguish "no" in the query: "a plain blue shirt with no logos". Prior work has largely addressed this limitation through data-centric approaches, fine-tuning CLIP on large-scale synthetic negation datasets. However, these efforts are commonly evaluated using retrieval-based metrics that cannot reliably reflect whether negation is actually understood. In this paper, we identify two key limitations of such evaluation metrics and investigate an alternative evaluation framework based on Multimodal LLMs-as-a-judge, which typically excel at understanding simple yes/no questions about image content, providing a fair evaluation of negation understanding in CLIP models. We then ask whether there already exists a direction in the CLIP embedding space associated with negation. We find evidence that such a direction exists, and show that it can be manipulated through test-time intervention via representation engineering to steer CLIP toward negation-aware behavior without any fine-tuning. Finally, we test negation understanding on non-common image-text samples to evaluate generalization under distribution shifts.

Related papers

• The Sound of Absence: Audio-Language Embedding Models Struggle with Negation [58.04745279785462]
  We show that embedding models fail to encode negated sound concepts.<n>We introduce NegEval-Audio, a framework that converts datasets into two negation-aware tasks.
  arXiv  Detail & Related papers  (2026-07-14T02:55:00Z)
• What "Not" to Detect: Negation-Aware VLMs via Structured Reasoning and Token Merging [42.41372222021938]
  State-of-the-art vision-language models (VLMs) suffer from a critical failure in understanding negation, often referred to as affirmative bias.<n>We introduce CoVAND, a dataset constructed with a systematic chain-of-thought (CoT) and VQA-based pipeline to generate high-quality, instance-grounded negation data.<n>Second, we propose NegToMe, a novel text token merging module that directly tackles the architectural cause of affirmative bias.
  arXiv  Detail & Related papers  (2025-10-15T07:36:38Z)
• Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding [4.9301587184653295]
  Negation is a fundamental linguistic phenomenon that poses persistent challenges for Large Language Models.<n>Existing benchmarks often treat negation as a side case within broader tasks like natural language inference.<n>We introduce Thunder-NUBench, a novel benchmark explicitly designed to assess sentence-level negation understanding in LLMs.
  arXiv  Detail & Related papers  (2025-06-17T10:51:39Z)
• Know "No" Better: A Data-Driven Approach for Enhancing Negation Awareness in CLIP [57.33324843049638]
  We introduce data generation pipelines that employ a large language model (LLM) and a multimodal LLM to produce negation-inclusive captions.<n>Fine-tuning CLIP with data generated from our pipelines, we develop NegationCLIP, which enhances negation awareness while preserving the generality.<n>Experiments on various CLIP architectures validate the effectiveness of our data generation pipelines in enhancing CLIP's ability to perceive negation accurately.
  arXiv  Detail & Related papers  (2025-01-19T01:17:05Z)
• Vision-Language Models Do Not Understand Negation [50.27667000027403]
  NegBench is a benchmark designed to evaluate negation understanding across 18 task variations and $79$k examples.<n>We show that this approach can result in a 10% increase in recall on negated queries and a 28% boost in accuracy on multiple-choice questions with negated captions.
  arXiv  Detail & Related papers  (2025-01-16T09:55:42Z)
• Revisiting subword tokenization: A case study on affixal negation in large language models [57.75279238091522]
  We measure the impact of affixal negation on modern English large language models (LLMs) We conduct experiments using LLMs with different subword tokenization methods. We show that models can, on the whole, reliably recognize the meaning of affixal negation.
  arXiv  Detail & Related papers  (2024-04-03T03:14:27Z)
• Language models are not naysayers: An analysis of language models on negation benchmarks [58.32362243122714]
  We evaluate the ability of current-generation auto-regressive language models to handle negation. We show that LLMs have several limitations including insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason under negation.
  arXiv  Detail & Related papers  (2023-06-14T01:16:37Z)
• Not another Negation Benchmark: The NaN-NLI Test Suite for Sub-clausal Negation [59.307534363825816]
  Negation is poorly captured by current language models, although the extent of this problem is not widely understood. We introduce a natural language inference (NLI) test suite to enable probing the capabilities of NLP methods.
  arXiv  Detail & Related papers  (2022-10-06T23:39:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.