HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

• URL: http://arxiv.org/abs/2607.00428v1
• Date: Wed, 01 Jul 2026 04:40:11 GMT
• Title: HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding
• Abstract summary: HyFL-CLIP is a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space.<n>Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation.
• Score: 28.162977602129303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on short captions. In long contexts, sentences are often reordered, summarized, or partially omitted. Although prior works extend CLIP with longer positional encodings, they often suffer from degraded image-text alignment under such text perturbations. We attribute this limitation to the Euclidean contrastive objective, which enforces strict one-to-one matching and lacks explicit mechanisms for modeling hierarchical relationships between global context and its constituent elements. To address this issue, we propose HyFL-CLIP, a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space via cross-manifold similarity distillation, leveraging its geometry to capture hierarchical and entailment relations. Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation. Experiments on diverse benchmarks, including long-context cross-modal retrieval, cross-modal retrieval with caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval, show that HyFL-CLIP achieves more robust long-context understanding. In particular, it yields up to 19.5% improvement in long-text cross-modal retrieval under textual perturbations over the best prior method. We also show HyFL-CLIP can be seamlessly integrated into other model frameworks by applying it to Stable Diffusion XL (SDXL).

Related papers

• Fine-grained CLIP fine-tuning with self-annotated region alignment [57.28808560528696]
  Contrastive Language-Image Pre-training has been shown to have limitations in its fine-grained dense feature representation.<n>We propose SFF-CLIP (Self-annotated Fine-grained Fine-tuning for CLIP) to boost the fine-grained representation ability in the CLIP fine-tuning.
  arXiv  Detail & Related papers  (2026-07-15T10:06:39Z)
• CLIP Is Shortsighted: Paying Attention Beyond the First Sentence [13.197160495980318]
  We introduce DeBias-CLIP, which removes the summary sentence during training and applies sentence sub-sampling and text token padding to distribute supervision across all token positions.<n>DeBias-CLIP achieves state-of-the-art long-text retrieval, improves short-text retrieval, and is less sensitive to sentence order permutations.
  arXiv  Detail & Related papers  (2026-02-25T21:27:00Z)
• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• $β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment [53.42377319350806]
  $$-CLIP is a multi-granular text-conditioned contrastive learning framework.<n>$$-CAL addresses the semantic overlap inherent in this hierarchy.<n>$$-CLIP establishes a robust, adaptive baseline for dense vision-language correspondence.
  arXiv  Detail & Related papers  (2025-12-14T13:03:20Z)
• MulCLIP: A Multi-level Alignment Framework for Enhancing Fine-grained Long-context CLIP [4.6096940605642915]
  MulCLIP is an end-to-end framework that bridges natural long-text structures with image components.<n>It preserves global contrastive alignment between images and both summary and long captions.<n>It extends positional embeddings for longer text sequences.
  arXiv  Detail & Related papers  (2025-12-08T03:23:41Z)
• HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment [13.584710249222105]
  HiMo-CLIP is a representation-level framework that enhances CLIP-style models without modifying the encoder architecture.<n>HiMo-CLIP introduces two key components: a hierarchical decomposition (HiDe) module that extracts latent semantic components from long-form text via in-batch PCA, and a monotonicity-aware contrastive loss (MoLo)<n>Experiments on multiple image-text retrieval benchmarks show that HiMo-CLIP consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-11-10T03:04:36Z)
• FineLIP: Extending CLIP's Reach via Fine-Grained Alignment with Longer Text Inputs [0.351124620232225]
  FineLIP enhances cross-modal text-image mapping by incorporating textbfFine-grained alignment with textbfLonger text input.<n>FineLIP first extends the positional embeddings to handle longer text, followed by the dynamic aggregation of local image and text tokens.<n>We validate our model on datasets with long, detailed captions across two tasks: zero-shot cross-modal retrieval and text-to-image generation.
  arXiv  Detail & Related papers  (2025-04-02T17:19:59Z)
• TULIP: Token-length Upgraded CLIP [57.818513403100326]
  We address the challenge of representing long captions in vision-language models, such as CLIP.<n>By design these models are limited by fixed, absolute positional encodings, restricting inputs to a maximum of 77 tokens.<n>We propose a generalizable method, named T, able to upgrade the token length to any length for CLIP-like models.
  arXiv  Detail & Related papers  (2024-10-13T22:34:15Z)
• Symmetrical Linguistic Feature Distillation with CLIP for Scene Text Recognition [77.93678598476149]
  We establish a novel Symmetrical Linguistic Feature Distillation framework (named CLIP-OCR) By cascading the CLIP image encoder with the reversed CLIP text encoder, a symmetrical structure is built with an image-to-text feature flow. Extensive experiments demonstrate the effectiveness of CLIP-OCR with 93.8% average accuracy on six popular STR benchmarks.
  arXiv  Detail & Related papers  (2023-10-08T04:00:20Z)
• BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval [61.803481264081036]
  Content-Based Image Retrieval (CIR) aims to search for a target image by concurrently comprehending the composition of an example image and a complementary text. We tackle this task by a novel underlinetextbfBottom-up crunderlinetextbfOss-modal underlinetextbfSemantic compounderlinetextbfSition (textbfBOSS) with Hybrid Counterfactual Training framework.
  arXiv  Detail & Related papers  (2022-07-09T07:14:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.