What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features

• URL: http://arxiv.org/abs/2607.23271v1
• Date: Sat, 25 Jul 2026 16:15:50 GMT
• Title: What CLIP Knows but Cannot Say: Recovering Negation from Frozen Intermediate Features
• Abstract summary: Contrastive vision-language models such as CLIP map semantically opposite phrases to nearly identical embeddings.<n>We show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises.<n>We propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak.
• Score: 3.9492850032415876
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Contrastive vision-language models such as CLIP map semantically opposite phrases (e.g., "a dog" vs. "not a dog") to nearly identical embeddings, rendering them insensitive to negation. We attribute this failure to a phenomenon we call Representational Collapse: by tracking compositional divergence and visual alignment across the CLIP text encoder, we show that middle layers build compositional syntax, but the final layers collapse this structure as visual alignment rises, producing a syntax-blind final representation. To recover the lost negation signal without altering pretrained weights, we propose PeakPatch, a lightweight post-hoc correction system that intercepts the encoder at its compositional peak. An Embedding Correction Network (ECN) uses cross-attention to extract a negation-specific signal from the peak layer, anchored to a stable baseline, and predicts a deviation vector that re-injects the lost syntax into the final-layer embedding space. A complementary Score Correction Network (SCN) predicts bounded scalar score offsets for discriminative tasks. Both modules are trained jointly end-to-end while all CLIP parameters remain frozen, adding only 5.2M parameters (3.5% of the backbone) and preserving the standard cosine similarity interface. On NegBench, PeakPatch achieves 74.3% on COCO MCQ (+35.1 over CLIP, +17.8 over the best encoder fine-tuning method) and 65.5% on VOC MCQ, while outperforming all fine-tuning baselines on fully out-of-distribution negation retrieval despite training only 3.5% of the parameters. The corrected embeddings also transfer to text-to-image generation (+18.4 negation score) and generalize across ViT-B/32, ViT-L/14, and SigLIP backbones. Project URL: https://stevencylu.github.io/PeakPatch/.

Related papers

• Disparity Has a Sign: Stereo Matching Beyond the Zero-Disparity Plane [52.210414985125816]
  We show how a stereoscopic content contains objects behind the zero-disparity plane corresponding to negative disparities.<n>We propose textitZDPShift, a benchmark of $21,495$ stereo pairs from seven cinematographer-authored open movies.<n>On identical scene content, Foundation goes from $2.24$ EPE to $75.33$px, with every backbone leaving roughly half of all pixels exceeding a three-pixel disparity error.
  arXiv  Detail & Related papers  (2026-09-06T19:58:35Z)
• Can Scene Text Recognition Read Rare Compositions? [1.7844382164707184]
  Scene text recognition is reported as 89--97% accurate on the six standard benchmarks, and the problem is widely treated as saturated.<n>We present an alternative reading.<n>When the same test images are stratified jointly by ground-truth word rarity and character n-gram novelty against a reference corpus, accuracy at the rareword x rare-trigram corner of the resulting 5x5 grid drops 10--18 pt below the q3/q3 centre.
  arXiv  Detail & Related papers  (2026-09-01T07:16:42Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Which Negatives Matter? Ask Your Text Encoder: Adaptive Similarity Margins for Dense-Caption Retrieval [20.130842837418726]
  We introduce HN-CLIP, which uses the text encoder's own text-text geometry to construct per-negative adaptive similarity margins.<n>The resulting objective requires only one caption-similarity matrix and a masked logit addition during training.<n>Experiments show that HN-CLIP improves over the strongest competitors by +2.5 to +4.0 R@1 while training 2.4x faster than GOAL and 5.4x faster than StructXLIP.
  arXiv  Detail & Related papers  (2026-08-19T04:16:46Z)
• Lost in Visual Translation: A VLM-Assisted Perceptual-Semantic Coherence Framework for EEG-to-Image Reconstruction [0.10499611180329804]
  EEG-to-image evaluation should distinguish visual fidelity from recoverable meaning.<n>SSIM, LPIPS, and CLIP penalize semantically recoverable outputs or reward plausible but incorrect ones.<n>We analyze 6,855 ground-truth/reconstruction pairs from ATM, ENIGMA, BrainVis, and DreamDiffusion.
  arXiv  Detail & Related papers  (2026-07-14T05:40:23Z)
• Recovering Input Text from Hidden States: Study of Gradient-Based Inversion of Decoder-Only Language Models [0.0]
  We study the hidden-state inversion problem as a continuous embedding-space optimisation.<n>We analyse which tokens break the reconstructions and find a sharp categorical asymmetry.<n>Last-layer hidden states of GPT-2 are as sensitive as original text.
  arXiv  Detail & Related papers  (2026-07-01T12:18:12Z)
• Efficient Visual Pointing for Embodied AI:Agent-Driven Data Synthesis, Cross-Block Attention, and Iterative Correction [55.11480729304395]
  We describe our PointArena 2026 solution, which achieves 77.2% overall accuracy and ranks second on the benchmark.<n>The ap proach targets three failure modes. First, agent-driven syn thesis builds large semantic and anchor-relative candidate pools.<n>Second, a determinis tic steerable-data pipeline creates a verified 10,000-sample main set, plus reserve samples, using masks, templates, and path verification.
  arXiv  Detail & Related papers  (2026-06-29T06:39:03Z)
• What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining [5.739405014622565]
  CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals.<n>We show that this signal saturates: once coarse mismatches are removed, stricter global filtering no longer tracks the compositional supervision provided by the retained captions.<n>We address this with Counterfactual Phrase Intervention (CPI), a phrase-level curation framework that converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores.
  arXiv  Detail & Related papers  (2026-05-21T15:55:15Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• Half-Truths Break Similarity-Based Retrieval [21.70058524831519]
  When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong.<n>We show that CLIP-style dual encoders often violate this intuition.<n>We propose CS-CLIP, which decomposes captions into entity and relation units.
  arXiv  Detail & Related papers  (2026-02-27T10:56:31Z)
• Not Just What's There: Enabling CLIP to Comprehend Negated Visual Descriptions Without Fine-tuning [23.10421006625293]
  Vision-Language Models (VLMs) like CLIP struggle to understand negation.<n>Existing methods refine negation understanding via fine-tuning CLIP's text encoder, risking overfitting.<n>We propose CLIPGlasses, a plug-and-play framework that enhances CLIP's ability to comprehend negated visual descriptions.
  arXiv  Detail & Related papers  (2026-02-24T15:55:39Z)
• TripleFDS: Triple Feature Disentanglement and Synthesis for Scene Text Editing [56.73004765030206]
  Scene Text Editing (STE) aims to naturally modify text in images while preserving visual consistency.<n>We propose TripleFDS, a novel framework for STE with disentangled modular attributes.<n>TripleFDS achieves state-of-the-art image fidelity (SSIM of 44.54) and text accuracy (ACC of 93.58%) on the mainstream STE benchmarks.
  arXiv  Detail & Related papers  (2025-11-17T14:15:03Z)
• Frozen CLIP: A Strong Backbone for Weakly Supervised Semantic Segmentation [90.35249276717038]
  We propose WeCLIP, a CLIP-based single-stage pipeline, for weakly supervised semantic segmentation. Specifically, the frozen CLIP model is applied as the backbone for semantic feature extraction. A new decoder is designed to interpret extracted semantic features for final prediction.
  arXiv  Detail & Related papers  (2024-06-17T03:49:47Z)
• Composed Fine-Tuning: Freezing Pre-Trained Denoising Autoencoders for Improved Generalization [93.95299500688286]
  We focus on prediction problems with structured outputs subject to output validity constraints. We propose composed fine-tuning, which fine-tunes a predictor composed with the pre-trained denoiser. For two-layer ReLU networks, we prove that composed fine-tuning significantly reduces the complexity of the predictor.
  arXiv  Detail & Related papers  (2020-06-29T17:14:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.