Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Family

• URL: http://arxiv.org/abs/2604.05971v1
• Date: Tue, 07 Apr 2026 15:04:33 GMT
• Title: Is CLIP Cross-Eyed? Revealing and Mitigating Center Bias in the CLIP Family
• Abstract summary: We show that contrastive vision-language models such as CLIP often lack fine-grained understanding of visual content.<n>Specifically, CLIP tends to disproportionately focus on the central region of an image, overlooking important objects located near the boundaries.<n>We show that this bias can be alleviated with training-free strategies such as visual prompting and attention redistribution.
• Score: 14.941271473863466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent research has shown that contrastive vision-language models such as CLIP often lack fine-grained understanding of visual content. While a growing body of work has sought to address this limitation, we identify a distinct failure mode in the CLIP family, which we term center bias, that persists even in recent model variants. Specifically, CLIP tends to disproportionately focus on the central region of an image, overlooking important objects located near the boundaries. This limitation is fundamental as failure to recognize relevant objects makes it difficult to perform any sophisticated tasks that depend on those objects. To understand the underlying causes of the limitation, we conduct analyses from both representation and attention perspectives. Using interpretability methods, i.e., embedding decomposition and attention map analysis, we find that relevant concepts especially those associated with off-center objects vanish from the model's embedding in the final representation due to information loss during the aggregation of visual embeddings, particularly the reliance on pooling mechanisms. Finally, we show that this bias can be alleviated with training-free strategies such as visual prompting and attention redistribution by redirecting models' attention to off-center regions.

Related papers

• Metonymy in vision models undermines attention-based interpretability [31.26551387218204]
  Part-based reasoning is a strategy to make a computer vision model directly focus on the object parts that are relevant to the downstream task.<n>We show that modern pretrained vision transformers violate the locality assumption and exhibit a strong intra-object leakage.<n>We then show that this inherently disentangled feature extraction improves attribute-driven part discovery on a variety of tasks.
  arXiv  Detail & Related papers  (2026-05-07T12:14:46Z)
• MentisOculi: Revealing the Limits of Reasoning with Mental Imagery [63.285794947638614]
  We develop MentisOculi, a suite of multi-step reasoning problems amenable to visual solution.<n> evaluating visual strategies ranging from latent tokens to explicit generated imagery, we find they generally fail to improve performance.<n>Our findings suggest that despite their inherent appeal, visual thoughts do not yet benefit model reasoning.
  arXiv  Detail & Related papers  (2026-02-02T18:49:06Z)
• Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective [47.99651635870674]
  We propose a training-free approach that emulates human distraction-refocusing behavior to redirect attention from distraction tokens back to target regions.<n>Our method achieves SOTA performance on eight benchmarks while maintaining high inference efficiency.
  arXiv  Detail & Related papers  (2025-11-20T09:16:33Z)
• Generalized Decoupled Learning for Enhancing Open-Vocabulary Dense Perception [71.26728044621458]
  DeCLIP is a novel framework that enhances CLIP by decoupling the self-attention module to obtain content'' and context'' features respectively.<n>It consistently achieves state-of-the-art performance across a broad spectrum of tasks, including 2D detection and segmentation, 3D instance segmentation, video instance segmentation, and 6D object pose estimation.
  arXiv  Detail & Related papers  (2025-08-15T06:43:51Z)
• Debiasing CLIP: Interpreting and Correcting Bias in Attention Heads [29.880490526874876]
  We introduce textsc-Then-Correct (LTC), a contrastive framework that identifies spurious attention heads and mitigates them through targeted ablation.<n>We evaluate LTC on benchmarks with inherent background gender biases, achieving over a $>50%$ gain in worst-group accuracy compared to non-training post-hoc baselines.<n>We visualize the representation of selected heads and find that the presented interpretation corroborates our contrastive mechanism for identifying both spurious and salient attention heads.
  arXiv  Detail & Related papers  (2025-05-23T03:13:42Z)
• Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas [69.56484419619919]
  We study the spatial reasoning challenge from the lens of mechanistic interpretability.<n>We observe that successful spatial reasoning correlates strongly with the model's ability to align its attention with actual object locations.<n>Motivated by these findings, we propose ADAPTVIS to sharpen the attention on highly relevant regions when confident.
  arXiv  Detail & Related papers  (2025-03-03T17:57:03Z)
• Unveiling Glitches: A Deep Dive into Image Encoding Bugs within CLIP [0.0]
  We focus on CLIP, a model renowned for its integration of vision and language processing. Our objective is to uncover recurring problems and blind spots in CLIP's image comprehension. We reveal significant discrepancies in CLIP's interpretation of images compared to human perception.
  arXiv  Detail & Related papers  (2024-06-30T05:23:11Z)
• A Closer Look at the Explainability of Contrastive Language-Image Pre-training [16.10032166963232]
  Contrastive language-image pre-training (CLIP) is a powerful vision-language model that has shown great benefits for various tasks. We have identified some issues with its explainability, which undermine its credibility and limit the capacity for related tasks. We propose the CLIP Surgery for reliable CAM, a method that allows surgery-like modifications to the inference architecture and features.
  arXiv  Detail & Related papers  (2023-04-12T07:16:55Z)
• Knowledge-guided Causal Intervention for Weakly-supervised Object Localization [32.99508048913356]
  KG-CI-CAM is a knowledge-guided causal intervention method. We tackle the co-occurrence context confounder problem via causal intervention. We introduce a multi-source knowledge guidance framework to strike a balance between absorbing classification knowledge and localization knowledge.
  arXiv  Detail & Related papers  (2023-01-03T12:02:19Z)
• Self-Supervised Video Object Segmentation via Cutout Prediction and Tagging [117.73967303377381]
  We propose a novel self-supervised Video Object (VOS) approach that strives to achieve better object-background discriminability. Our approach is based on a discriminative learning loss formulation that takes into account both object and background information. Our proposed approach, CT-VOS, achieves state-of-the-art results on two challenging benchmarks: DAVIS-2017 and Youtube-VOS.
  arXiv  Detail & Related papers  (2022-04-22T17:53:27Z)
• Dual Contrastive Learning for General Face Forgery Detection [64.41970626226221]
  We propose a novel face forgery detection framework, named Dual Contrastive Learning (DCL), which constructs positive and negative paired data. To explore the essential discrepancies, Intra-Instance Contrastive Learning (Intra-ICL) is introduced to focus on the local content inconsistencies prevalent in the forged faces.
  arXiv  Detail & Related papers  (2021-12-27T05:44:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.