Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training

• URL: http://arxiv.org/abs/2608.29313v1
• Date: Sat, 29 Aug 2026 14:55:40 GMT
• Title: Hyper3-CLIP: Hierarchy-Conditioned Hyperbolic Vision-Language Training
• Abstract summary: Hyper3-CLIP is a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling.<n>It improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO.
• Score: 14.328379791611797
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: CLIP-like vision-language models (VLMs) trained with contrastive objectives learn strong global image-text representations, but their Euclidean embeddings and global pooling fail to encode relational structure such as part-whole and parent-child relations. Hyperbolic VLMs address this gap with entailment-based objectives, and text-conditioned variants improve fine-grained alignment through sentence- and phrase-level queries. However, these two lines of work remain separate: hyperbolic VLMs use static image and region features, while query-conditioned methods lack hierarchical geometric structure. We present Hyper3-CLIP, a hierarchy-conditioned hyperbolic VLM that combines global, local, and global-local contrastive learning with query-conditioned visual pooling. To train the model, we construct lightweight query hierarchies from text, comprising full captions, sentence fragments, localized part descriptions, and extracted phrases. Each query conditions the pooling of visual patches, and the resulting representations support image-text, whole-part, and parent-child entailment losses. Query-conditioned pooling is active only during training. Hyper3-CLIP improves R@5 and R@10 retrieval on COCO and Flickr, as well as multi-label classification on VOC and COCO, while remaining competitive on hierarchy metrics. We also audit zero-shot prompt sensitivity under fixed prompt regimes and study the effect of the localized GRIT part budget used during training. Code is available at https://github.com/Hyper3Labs/hyper3-clip.

Related papers

• HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding [28.162977602129303]
  HyFL-CLIP is a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space.<n>Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation.
  arXiv  Detail & Related papers  (2026-07-01T04:40:11Z)
• DVLA-RL: Dual-Level Vision-Language Alignment with Reinforcement Learning Gating for Few-Shot Learning [53.36809572236361]
  Few-shot learning aims to generalize to novel categories with only a few samples.<n>Recent approaches incorporate large language models to enrich visual representations with semantic embeddings derived from class names.<n>We propose Dual-level Vision-Language Alignment with Reinforcement Learning gating (DVLA-RL)
  arXiv  Detail & Related papers  (2026-01-31T16:09:37Z)
• SuperCLIP: CLIP with Simple Classification Supervision [88.86549733903314]
  Contrastive Language-Image Pretraining achieves strong generalization in vision-language tasks by aligning images and texts in a shared embedding space.<n>Recent findings show that CLIP-like models still underutilize fine-grained semantic signals in text.<n>We propose SuperCLIP, a framework that augments contrastive learning with classification-based supervision.
  arXiv  Detail & Related papers  (2025-12-16T15:11:53Z)
• $β$-CLIP: Text-Conditioned Contrastive Learning for Multi-Granular Vision-Language Alignment [53.42377319350806]
  $$-CLIP is a multi-granular text-conditioned contrastive learning framework.<n>$$-CAL addresses the semantic overlap inherent in this hierarchy.<n>$$-CLIP establishes a robust, adaptive baseline for dense vision-language correspondence.
  arXiv  Detail & Related papers  (2025-12-14T13:03:20Z)
• GRACE: Graph-Guided Repository-Aware Code Completion through Hierarchical Code Fusion [33.66085762717581]
  LLMs excel in localized code completion but struggle with repository-level tasks due to limited context windows.<n> GRACE builds a multi-level, multi-semantic code graph to capture both static and dynamic code semantics.<n>Experiments demonstrate that GRACE significantly outperforms state-of-the-art methods across all metrics.
  arXiv  Detail & Related papers  (2025-09-07T09:01:48Z)
• SmartCLIP: Modular Vision-language Alignment with Identification Guarantees [59.16312652369709]
  Contrastive Language-Image Pre-training (CLIP)citepradford2021learning has emerged as a pivotal model in computer vision and multimodal learning.<n>CLIP struggles with potential information misalignment in many image-text datasets and suffers from entangled representation.<n>We introduce ours, a novel approach that identifies and aligns the most relevant visual and textual representations in a modular manner.
  arXiv  Detail & Related papers  (2025-07-29T22:26:20Z)
• ReferGPT: Towards Zero-Shot Referring Multi-Object Tracking [17.736434513456576]
  ReferGPT is a novel zero-shot referring multi-object tracking framework.<n>We provide a multi-modal large language model (MLLM) with spatial knowledge enabling it to generate 3D-aware captions.<n>We also propose a robust query-matching strategy, leveraging CLIP-based semantic encoding and fuzzy matching to associate MLLM generated captions with user queries.
  arXiv  Detail & Related papers  (2025-04-12T12:33:15Z)
• TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives [65.82577305915643]
  Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. We show that generating hard'' negative captions via in-context learning and corresponding negative images with text-to-image generators offers a solution. We demonstrate that our method, named TripletCLIP, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark.
  arXiv  Detail & Related papers  (2024-11-04T19:24:59Z)
• Towards Realistic Zero-Shot Classification via Self Structural Semantic Alignment [53.2701026843921]
  Large-scale pre-trained Vision Language Models (VLMs) have proven effective for zero-shot classification. In this paper, we aim at a more challenging setting, Realistic Zero-Shot Classification, which assumes no annotation but instead a broad vocabulary. We propose the Self Structural Semantic Alignment (S3A) framework, which extracts structural semantic information from unlabeled data while simultaneously self-learning.
  arXiv  Detail & Related papers  (2023-08-24T17:56:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.