KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale

• URL: http://arxiv.org/abs/2607.18885v1
• Date: Tue, 21 Jul 2026 09:14:33 GMT
• Title: KALE: Kernel Alignment with Loss Equilibration for Stable CLIP-DINOv2 Alignment at Web Scale
• Abstract summary: Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility.<n>We introduce KALE, a loss-equilibration controller that tracks both losses and adaptively rescales the alignment weight toward a target ratio.<n>On a 3.3M-image CC12M subset, the aligned model preserves image-text retrieval and reproducibly improves SVHN linear probing.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Kernel-based alignment of CLIP toward a vision centric teacher such as DINOv2 (KUEA) improves CLIP's visual representations while preserving text-encoder compatibility, using a fixed trade-off weight tuned on curated ImageNet-1K. We ask whether this transfers to noisy, web-scale data (CC12M) and find that it does not: the alignment term's weighted contribution falls to about 0.2% of the clean term, so under any fixed weight its gradient is effectively inert. We introduce KALE, a loss-equilibration controller that tracks both losses and adaptively rescales the alignment weight toward a target ratio, restoring the signal with no per-dataset tuning; reaching balance requires increasing the weight by roughly four orders of magnitude, and the required value is configuration-dependent, so no fixed scalar suffices. We characterize the resulting regime: a bounded high learning rate and a decaying schedule with a moderate floor are needed for stability, and the controller equilibrates rather than diverging. On a 3.3M-image CC12M subset, the aligned model preserves image-text retrieval and reproducibly improves SVHN linear probing; zero-shot improves by +2.00 over CLIP on the standard 11-dataset average, exceeding KUEA's +1.29. We report all results with explicit run-to-run variance and base our conclusions on the metrics that are stable across runs.

Related papers

• COEC: Calibrated Orthogonal-Equivalence Compensation for Structured Pruning of Large Language Models [13.501861808796873]
  We propose COEC (Calibrated Orthogonal-Equivalence Compensation) as a training-free compensation framework for structured pruning.<n>Experiments on the Llama-3, Llama-3.1, and Qwen2.5 model families show that COEC improves perplexity on every model and zero-shot accuracy in most settings.
  arXiv  Detail & Related papers  (2026-08-21T14:20:49Z)
• SPARC-Net: A Spectral, Causality-Aware, and Hard-Constrained Physics-Informed Architecture for Stiff and Shock-Dominated Partial Differential Equations [0.0654051537828115]
  Physics-Informed Neural Networks (PINNs) provide a meshless approach for solving partial differential equations (PDEs)<n>PINNs suffer severe degradation in stiff and shock-dominated problems, where small PDE residuals can correspond to globally inaccurate solutions.<n>We show these failures are multi-causal, arising from the concurrent interplay of (i) spectral bias against sharp features, (ii) imbalanced multi-term optimization and loss-weight collapse, (iii) violation of temporal causality, and (iv) under-resolved collocation.<n>We present-Net, a unified architecture and training
  arXiv  Detail & Related papers  (2026-07-13T09:23:17Z)
• CaliDist: Calibrating Large Language Models via Behavioral Robustness to Distraction [51.56484100374058]
  Existing calibration methods for Large Language Models (LLMs) often overlook a critical dimension of trustworthiness: a model's behavioral robustness.<n>We introduce textscCaliDist, a novel post-hoc calibration approach that directly measures and penalizes a model's susceptibility to distraction.<n>textscCaliDist consistently achieves lower Expected Error (ECE) and Brier Score compared with strong baselines.
  arXiv  Detail & Related papers  (2026-06-04T07:27:53Z)
• Attention Calibration for Position-Fair Dense Information Retrieval [29.915108607793996]
  Dense retrieval models exhibit positional bias when relevant information appears later in a passage.<n>We ask whether this bias can be reduced at inference time, without retraining and without sacrificing overall retrieval effectiveness.<n>We adapt inference-time attention calibration to downstream retrieval and extend it with a strength coefficient that interpolates between the original and fully calibrated attention distributions.
  arXiv  Detail & Related papers  (2026-06-01T18:04:26Z)
• VISReg: Variance-Invariance-Sketching Regularization for JEPA training [21.894781659564817]
  Self-supervised learning methods prevent embedding collapse via modelings or explicit regularization of the embedding space.<n>We propose Variance-Invariance-Sketching Regularization (VISReg)<n>VISReg replaces covariance with a Sliced-Wasserstein-based sketching objective that enforces full distributional shape.<n>We show that VISReg scales linearly, outperforms existing regularization on low-quality datasets, and is resilient to long-tailed and low-rank regimes.
  arXiv  Detail & Related papers  (2026-06-01T17:58:01Z)
• Distortion-Aware Fusion of Statistical and Vision-Language Features for Blind Image Quality Assessment [3.4153424359998508]
  We propose a distortion-aware fusion framework that integrates a 138-dimensional NSS descriptor with two complementary VLM embeddings.<n>A per-distortion analysis on KADID-10k reveals that NSS features contribute most on noise and color-shift distortions.
  arXiv  Detail & Related papers  (2026-06-01T09:57:52Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Rethinking Long-tailed Dataset Distillation: A Uni-Level Framework with Unbiased Recovery and Relabeling [105.8570596633629]
  We rethink long-tailed dataset distillation by revisiting the limitations of trajectory-based methods.<n>We adopt the statistical alignment perspective to jointly model bias and restore fair supervision.<n>Our approach improves top-1 accuracy by 15.6% on CIFAR-100-LT and 11.8% on Tiny-ImageNet-LT.
  arXiv  Detail & Related papers  (2025-11-24T07:57:01Z)
• Cautious Weight Decay [23.198565281737896]
  Cautious Weight Decay (CWD) is a one-line, agnostic modification that applies weight decay only to parameter whose signs align with update.<n>CWD is a drop-in change for coordinates such as AdamW, Lion and Muon.<n>For language model pre-training and ImageNet classification, CWD consistently improves final loss and accuracy.
  arXiv  Detail & Related papers  (2025-10-14T11:32:55Z)
• Hyperspherical Normalization for Scalable Deep Reinforcement Learning [57.016639036237315]
  SimbaV2 is a novel reinforcement learning architecture designed to stabilize optimization.<n>It scales up effectively with larger models and greater compute, achieving state-of-the-art performance on 57 continuous control tasks.
  arXiv  Detail & Related papers  (2025-02-21T08:17:24Z)
• Semi-Supervised Coupled Thin-Plate Spline Model for Rotation Correction and Beyond [84.56978780892783]
  We propose CoupledTPS, which iteratively couples multiple TPS with limited control points into a more flexible and powerful transformation. In light of the laborious annotation cost, we develop a semi-supervised learning scheme to improve warping quality by exploiting unlabeled data. Experiments demonstrate the superiority and universality of CoupledTPS over the existing state-of-the-art solutions for rotation correction.
  arXiv  Detail & Related papers  (2024-01-24T13:03:28Z)
• Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity [55.29408396918968]
  We study a family of loss functions named label-distributionally robust (LDR) losses for multi-class classification. Our contributions include both consistency and robustness by establishing top-$k$ consistency of LDR losses for multi-class classification. We propose a new adaptive LDR loss that automatically adapts the individualized temperature parameter to the noise degree of class label of each instance.
  arXiv  Detail & Related papers  (2021-12-30T00:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.