Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance

• URL: http://arxiv.org/abs/2609.05730v1
• Date: Fri, 04 Sep 2026 21:29:20 GMT
• Title: Bigger Text Encoders Can Hurt CLIP Zero-Shot Performance
• Abstract summary: We show that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades.<n>Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters.
• Score: 23.886374573928965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive Language-Image Pretraining (CLIP) is a building block of many machine learning applications. Scaling laws have guided resource allocation for large-scale training, yet prior work treats total CLIP model size as a single variable, without exploring how the capacity split between encoders impacts downstream performance. Here, we train multiple CLIP models with different vision and text encoder sizes, revealing that for most vision encoders, there is an optimal text encoder size beyond which zero-shot performance degrades---even as total parameter count increases. Exploiting this behavior yields efficient configurations that match the zero-shot performance of the standard ViT-B/16 architecture with up to 55% fewer parameters. We further show that this degradation stems from overfitting induced by the oversized text encoder, and that using modality-specific weight decay coefficients not only recovers but improves performance across all degraded configurations. A geometric analysis reveals a trade-off in which scaling the text encoder improves embedding uniformity but worsens cross-modal alignment; we further show that these metrics are predictive of zero-shot performance. We hope these findings motivate CLIP architectures and training methods that counteract this degradation, a prerequisite for scaling CLIP reliably and efficiently.

Related papers

• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• Data or Language Supervision: What Makes CLIP Better than DINO? [50.59472280781008]
  We show that CLIP captures high-level semantics, while DINO is more responsive to low-level features like colors and styles.<n>When integrated into vision-language models, CLIP excels at text-intensive tasks, while DINO slightly outperforms on vision-centric ones.
  arXiv  Detail & Related papers  (2025-10-13T18:34:58Z)
• Robustness in Both Domains: CLIP Needs a Robust Text Encoder [55.348860778166426]
  LEAF is an efficient adversarial finetuning method for the text domain, with the ability to scale to large CLIP models.<n>Our models significantly improve the zero-shot adversarial accuracy in the text domain, while maintaining the vision performance provided by robust image encoders.<n>We show that robust text encoders facilitate better reconstruction of input text from its embedding via direct optimization.
  arXiv  Detail & Related papers  (2025-06-03T19:57:09Z)
• AmorLIP: Efficient Language-Image Pretraining via Amortization [52.533088120633785]
  Contrastive Language-Image Pretraining (CLIP) has demonstrated strong zero-shot performance across diverse downstream text-image tasks.<n>We propose AmorLIP, an efficient CLIP pretraining framework that amortizes expensive computations involved in contrastive learning through lightweight neural networks.
  arXiv  Detail & Related papers  (2025-05-25T05:30:37Z)
• Image Coding for Machines via Feature-Preserving Rate-Distortion Optimization [31.210700220124192]
  We show that an approach to reduce the effect of compression for a given task loss is to perform rate-distortion (RDO) using the distance between features.<n>We simplify the RDO formulation to make the distortion term computable using block-based encoders.<n>We show that simulations with transformed HEVC across multiple feature extractors and downstream networks show up to 17 % bit-rate savings for the same task accuracy.
  arXiv  Detail & Related papers  (2025-04-03T02:11:26Z)
• DiffCLIP: Differential Attention Meets CLIP [57.396578974401734]
  We propose DiffCLIP, a novel vision-language model that extends the differential attention mechanism to CLIP architectures.<n>With minimal additional parameters, DiffCLIP achieves superior performance on image-text understanding tasks.
  arXiv  Detail & Related papers  (2025-03-09T14:04:09Z)
• CLIP's Visual Embedding Projector is a Few-shot Cornucopia [45.93202559299953]
  We introduce an alternative way for few-shot CLIP adaptation without adding ''external'' parameters to optimize.<n>We find that simply fine-tuning the embedding projection matrix of the vision leads to better performance than all baselines.<n>This simple approach, coined ProLIP, yields state-of-the-art performance on 11 few-shot classification benchmarks, few-shot cross-dataset encoder transfer, domain generalization, and base-to-new class generalization.
  arXiv  Detail & Related papers  (2024-10-07T17:59:59Z)
• Progressive Token Length Scaling in Transformer Encoders for Efficient Universal Segmentation [67.85309547416155]
  A powerful architecture for universal segmentation relies on transformers that encode multi-scale image features and decode object queries into mask predictions.<n>With efficiency being a high priority for scaling such models, we observed that the state-of-the-art method Mask2Former uses 50% of its compute only on the transformer encoder.<n>This is due to the retention of a full-length token-level representation of all backbone feature scales at each encoder layer.
  arXiv  Detail & Related papers  (2024-04-23T01:34:20Z)
• Extreme Encoder Output Frame Rate Reduction: Improving Computational Latencies of Large End-to-End Models [59.57732929473519]
  We apply multiple frame reduction layers in the encoder to compress encoder outputs into a small number of output frames. We demonstrate that we can generate one encoder output frame for every 2.56 sec of input speech, without significantly affecting word error rate on a large-scale voice search task.
  arXiv  Detail & Related papers  (2024-02-27T03:40:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.