DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency

• URL: http://arxiv.org/abs/2605.06592v1
• Date: Thu, 07 May 2026 17:19:52 GMT
• Title: DINORANKCLIP: DINOv3 Distillation and Injection for Vision-Language Pretraining with High-Order Ranking Consistency
• Abstract summary: Contrastive language-image preCLIP suffers from two structural weaknesses.<n>RANKCLIP partially addresses the first issue with a list-wise Plackett-Luce ranking-consistency loss.<n>We propose DINORANKCLIP, a pretraining framework that addresses both jointly.
• Score: 12.06248193140353
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive language-image pretraining (CLIP) suffers from two structural weaknesses: the symmetric InfoNCE loss discards the relative ordering among unmatched in-batch pairs, and global pooling collapses the visual representation into a semantic bottleneck that is poorly sensitive to fine-grained local structure. RANKCLIP partially addresses the first issue with a list-wise Plackett-Luce ranking-consistency loss, but its model is strictly first-order and inherits the second weakness untouched. We propose DINORANKCLIP, a pretraining framework that addresses both jointly. Our principal contribution is injecting a frozen DINOv3 teacher into the contrastive trunk through a dual-branch lightweight student and a multi-scale fusion module with channel-spatial attention, a self-attention refiner, and a conflict-aware gate that preserves the cross-modal alignment up to first order. Complementarily, we introduce a high-order Plackett-Luce ranking model in which the per-position utility is augmented with attention-parameterised pairwise and tuple-wise transition terms; the family contains CLIP and RANKCLIP as nested zero-order and first-order special cases, and the optimal order on every benchmark is $R^*=3$. The full empirical study -- order sweep, Fine-grained Probe on five datasets, four-node Modality-Gap analysis, six-variant Fusion ablation -- fits in 72 hours on a single eight-GPU H100 node and trains entirely on Conceptual Captions 3M. DINORANKCLIP consistently outperforms CLIP, CyCLIP, ALIP, and RANKCLIP under matched compute, with the largest relative gains on the fine-grained and out-of-distribution evaluations that most directly stress local structural reasoning.

Related papers

• Learning Where Outcomes Change:Credit-Addressable Reasoning for Multimodal Geometry [105.89493240787861]
  We introduce credit-addressable reasoning, in which semantic units exposed during inference also define where learning compares alternatives and assigns credit.<n>We instantiate this principle with Code-CoT, which represents visual relations as line-addressable executable code, and organizes reasoning into typed events.<n>Across nine geometry benchmarks, CE-GRPO achieves an average accuracy of 76.04, outperforming Qwen3-VL-8B and trajectory-level GRPO by $8.09$ and 3.43 points, respectively.
  arXiv  Detail & Related papers  (2026-08-31T08:44:36Z)
• Hyper-Fold: Exploring the Expressive Limit of Sequence-Geometry Learning for Proteins via Hypergraph Modeling [55.36837305276515]
  We show that the complete bilinear operator over content-geometry outer products--the sufficient statistic of all second-order interactions--is the expressive ceiling.<n>We then introduce Hyper-Fold, a rank-K separable convolutional backbone approaching this ceiling at message-passing cost.
  arXiv  Detail & Related papers  (2026-08-29T11:43:25Z)
• Text-Guided Visual Dependency Graph Learning with Cross-Modal Attention Priors [11.338098263634537]
  We propose CM-GLasso, a framework that bridges vision-language representation learning and sparse Gaussian Models.<n> CM-GLasso introduces three key components: (i) a text visualization strategy that renders class-attribute descriptions as images and processes them through the same SigLIP-2 vision encoder as natural images, (ii) a cross-attention distillation mechanism that condenses high-dimensional patches into a small set of semantic graph nodes, and (iii) a joint ADMM formulation that estimates shared and class-specific precision components within a single convex objective.
  arXiv  Detail & Related papers  (2026-08-19T09:57:34Z)
• Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization [3.573729952898132]
  Production LLMs receive instructions from sources with very different levels of trust, yet attend to every token with uniform architectural privilege.<n>This is the structural vulnerability that enables malicious prompt injections and, more broadly, leaves models without a principled way to resolve conflicts between legitimate but competing instructions.<n>We introduce Gravity-Weighted DPO, a preference-optimization objective whose per-sample offset scales with the structural distance between conflicting levels under a linear or bilateral schedule.
  arXiv  Detail & Related papers  (2026-06-09T13:39:17Z)
• Convergence Theory for Iterative LLM-Based Neural Architecture Search: A Parametric Cross-Entropy Framework with Closed-Form Proxy Reliability [48.83701310501069]
  Large language models (LLMs) are increasingly used as generators in iterative neural architecture search (NAS)<n>We model iterative LLM-NAS as a parametric Cross-Entropy (CE) method over executable programs.<n>We prove six results: (1) iterative LLM fine-tuning on elite architectures is equivalent to the CE update restricted to the LLM family; (2) expected architecture quality is monotonically non-decreasing across cycles; and (3) elite-set probability converges to a fixed point at a geometric rate.
  arXiv  Detail & Related papers  (2026-05-28T15:45:19Z)
• Improving LLM Reasoning with Homophily-aware Structural and Semantic Text-Attributed Graph Compression [55.51959317490934]
  Large language models (LLMs) have demonstrated promising capabilities in Text-Attributed Graph (TAG) understanding.<n>We argue that graphs inherently contain rich structural and semantic information, and that their effective exploitation can unlock potential gains in LLMs reasoning performance.<n>We propose Homophily-aware Structural and Semantic Compression for LLMs (HS2C), a framework centered on exploiting graph homophily.
  arXiv  Detail & Related papers  (2026-01-13T03:35:18Z)
• Higher-order Linear Attention [59.92962330635185]
  quadratic cost of scaled dot-product attention is a central obstacle to scaling autoregressive language models to long contexts.<n>We introduce Higher-order Linear Attention (HLA), a causal, streaming mechanism that realizes higher interactions via compact prefix sufficient statistics.
  arXiv  Detail & Related papers  (2025-10-31T07:54:37Z)
• REALIGN: Regularized Procedure Alignment with Matching Video Embeddings via Partial Gromov-Wasserstein Optimal Transport [7.952582509792969]
  Real-world instructional data often contains background segments, repeated actions, and steps presented out of order.<n>We introduce REALIGN, a self-supervised framework for procedure learning based on Regularized Fused Partial Gromov-Wasserstein Optimal Transport (R-FPGWOT)<n>In contrast to KOT, our formulation jointly models visual correspondences and temporal relations under a partial alignment scheme.
  arXiv  Detail & Related papers  (2025-09-29T07:32:14Z)
• S2-Attention: Hardware-Aware Context Sharding Among Attention Heads [49.1454481007861]
  Sparse attention selectively attends to a subset of tokens in the context.<n>It remains unclear whether sparse attention can maintain the model's quality at a scale of today's large language models.<n>This paper presents Sparsely-Sharded(S2) Attention, a Triton library that provides kernel optimization for sparse attention customizable at both per-head and per-context-range levels.
  arXiv  Detail & Related papers  (2024-07-25T00:27:07Z)
• Parameter-efficient Fine-tuning in Hyperspherical Space for Open-vocabulary Semantic Segmentation [10.502680141980642]
  Open-vocabulary semantic segmentation seeks to label each pixel in an image with arbitrary text descriptions.<n> vision-language foundation models, especially CLIP, have emerged as powerful tools for acquiring open-vocabulary capabilities.<n>H-CLIP achieves new SOTA open-vocabulary semantic segmentation results while only requiring updating approximately 4% of the total parameters of CLIP.
  arXiv  Detail & Related papers  (2024-05-29T07:41:34Z)
• RankCLIP: Ranking-Consistent Language-Image Pretraining [7.92247304974314]
  RankCLIP is a novel pre-training method that extends beyond the rigid one-to-one matching framework of CLIP.<n>By extending the traditional pair-wise loss to list-wise, RankCLIP improves the alignment process, enabling it to capture the nuanced many-to-many relationships between and within each modality.
  arXiv  Detail & Related papers  (2024-04-15T00:12:27Z)
• SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger [30.758184720183106]
  We propose SoftCLIP, a novel approach that relaxes the strict one-to-one constraint and achieves a soft cross-modal alignment. In particular, on ImageNet zero-shot classification task, using CC3M/CC12M as pre-training dataset, SoftCLIP brings a top-1 accuracy improvement of 6.8%/7.2%.
  arXiv  Detail & Related papers  (2023-03-30T17:27:22Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Generalized Zero-Shot Learning Via Over-Complete Distribution [79.5140590952889]
  We propose to generate an Over-Complete Distribution (OCD) using Conditional Variational Autoencoder (CVAE) of both seen and unseen classes. The effectiveness of the framework is evaluated using both Zero-Shot Learning and Generalized Zero-Shot Learning protocols.
  arXiv  Detail & Related papers  (2020-04-01T19:05:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.