Half-Truths Break Similarity-Based Retrieval

• URL: http://arxiv.org/abs/2602.23906v1
• Date: Fri, 27 Feb 2026 10:56:31 GMT
• Title: Half-Truths Break Similarity-Based Retrieval
• Abstract summary: When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong.<n>We show that CLIP-style dual encoders often violate this intuition.<n>We propose CS-CLIP, which decomposes captions into entity and relation units.
• Score: 21.70058524831519
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP

Related papers

• What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features [3.9492850032415876]
  Contrastive vision-language models such as CLIP map semantically opposite phrases to nearly identical embeddings.<n>We show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises.<n>We propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak.
  arXiv  Detail & Related papers  (2026-07-25T16:15:50Z)
• DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency [12.06248193140353]
  Contrastive language-image preCLIP suffers from two structural weaknesses.<n>RANKCLIP partially addresses the first issue with a list-wise Plackett-Luce ranking-consistency loss.<n>We propose DINORANKCLIP, a pretraining framework that addresses both jointly.
  arXiv  Detail & Related papers  (2026-05-07T17:19:52Z)
• Compositional Multi-hop Factual Error Correction via Decomposition-and-Injection [60.34509125517578]
  We present CECoR (Compositional Error Correction via Reasoning-aware Synthesis), a reasoning-aware framework for compositional error correction.<n> CECoR decomposes multi-hop claims into interpretable reasoning steps and injects controlled perturbations to synthesize high-quality training pairs.<n>A two-stage learning strategy combining supervised fine-tuning and reinforcement learning improves factual accuracy and robustness.
  arXiv  Detail & Related papers  (2026-05-04T07:08:39Z)
• MEMENTO: Teaching LLMs to Manage Their Own Context [50.3558738319336]
  Reasoning models think in long, unstructured streams with no mechanism for compressing or organizing their own intermediate state.<n>We introduce MEMENTO: a method that teaches models to segment reasoning into blocks, compress each block into a memento, and reason forward by attending only to mementos.<n>We show that a two-stage SFT recipe on OpenMementos is effective across different model families and scales.
  arXiv  Detail & Related papers  (2026-04-10T19:30:29Z)
• The Specification Gap: Coordination Failure Under Partial Knowledge in Code Agents [0.0]
  Two-agent integration accuracy drops from 58% to 25% as detail is removed.<n>A factorial recovery experiment shows that restoring the full specification alone recovers the single-agent ceiling.<n>The gap is not merely a consequence of hidden information, but reflects the difficulty of producing compatible code without shared decisions.
  arXiv  Detail & Related papers  (2026-03-25T13:18:26Z)
• Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning [23.10421006625293]
  Vision-Language Models (VLMs) like CLIP struggle to understand negation.<n>Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting.<n>We propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions.
  arXiv  Detail & Related papers  (2026-02-24T15:55:39Z)
• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• Language as a Label: Zero-Shot Multimodal Classification of Everyday Postures under Data Scarcity [0.764671395172401]
  Recent Vision-Language Models (VLMs) enable zero-shot classification by aligning images and text in a shared space.<n>This study investigates how prompt specificity affects the zero-shot classification of sitting, standing, and walking/running.
  arXiv  Detail & Related papers  (2025-10-15T09:53:46Z)
• L-CLIPScore: a Lightweight Embedding-based Captioning Metric for Evaluating and Training [40.87934127615281]
  We propose a novel embedding-based captioning metric computation termed as L-CLIPScore.<n>L-CLIPScore is calculated from a lightweight CLIP, which is a dual-encoder architecture compressed and distilled from CLIP.<n>We carry out exhaustive experiments to validate the efficiency and effectiveness of L-CLIPScore when using it as the judge to evaluate caption quality.
  arXiv  Detail & Related papers  (2025-07-11T16:08:12Z)
• Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting [86.15347226865826]
  We design a new end-to-end object-aware lifting approach, named Unified-Lift.<n>We augment each Gaussian point with an additional Gaussian-level feature learned using a contrastive loss to encode instance information.<n>We conduct experiments on three benchmarks: LERF-Masked, Replica, and Messy Rooms.
  arXiv  Detail & Related papers  (2025-03-18T08:42:23Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Learn "No" to Say "Yes" Better: Improving Vision-Language Models via Negations [43.484570564890866]
  Existing vision-language models (VLMs) treat text descriptions as a unit, confusing individual concepts in a prompt. We present CC-Neg, a dataset containing 228,246 images, true captions and their corresponding negated captions. Using CC-Neg along with modifications to the contrastive loss of CLIP, our proposed CoN-CLIP framework, has an improved understanding of negations.
  arXiv  Detail & Related papers  (2024-03-29T17:33:42Z)
• FiGCLIP: Fine-Grained CLIP Adaptation via Densely Annotated Videos [19.08882495584709]
  We show that it is possible to enhance CLIP's fine-grained and syntactic abilities without compromising its semantic properties. We adapt CLIP efficiently on a high-quality, comprehensive, and relatively small dataset. We learn a powerful visual representation, dubbed Fine-Grained CLIP (FiGCLIP), that preserves semantic understanding while being detail-oriented.
  arXiv  Detail & Related papers  (2024-01-15T13:27:34Z)
• Vita-CLIP: Video and text adaptive CLIP via Multimodal Prompting [111.49781716597984]
  We propose a multimodal prompt learning scheme that works to balance the supervised and zero-shot performance under a single unified training. We can achieve state-of-the-art zero-shot performance on Kinetics-600, HMDB51 and UCF101 while remaining competitive in the supervised setting.
  arXiv  Detail & Related papers  (2023-04-06T18:00:04Z)
• ComCLIP: Training-Free Compositional Image and Text Matching [19.373706257771673]
  Contrastive Language-Image Pretraining has demonstrated great zero-shot performance for matching images and text. We propose a novel textbftextittraining-free compositional CLIP model (ComCLIP) ComCLIP disentangles input images into subjects, objects, and action sub-images and composes CLIP's vision encoder and text encoder to perform evolving matching over compositional text embedding and sub-image embeddings.
  arXiv  Detail & Related papers  (2022-11-25T01:37:48Z)
• Improving Factual Consistency in Summarization with Compression-Based Post-Editing [146.24839415743358]
  We show that a model-agnostic way to address this problem is post-editing the generated summaries. We propose to use sentence-compression data to train the post-editing model to take a summary with extrinsic entity errors marked with special tokens. We show that this model improves factual consistency while maintaining ROUGE, improving entity precision by up to 30% on XSum, and that this model can be applied on top of another post-editor.
  arXiv  Detail & Related papers  (2022-11-11T13:35:38Z)
• Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization [93.95299500688286]
  We focus on prediction problems with structured outputs subject to output validity constraints. We propose composed fine-tuning, which fine-tunes a predictor composed with the pre-trained denoiser. For two-layer ReLU networks, we prove that composed fine-tuning significantly reduces the complexity of the predictor.
  arXiv  Detail & Related papers  (2020-06-29T17:14:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.