MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data

• URL: http://arxiv.org/abs/2603.08202v1
• Date: Mon, 09 Mar 2026 10:29:50 GMT
• Title: MM-TS: Multi-Modal Temperature and Margin Schedules for Contrastive Learning with Long-Tail Data
• Abstract summary: We propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning.<n>Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting.
• Score: 64.78447637450937
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning has become a fundamental approach in both uni-modal and multi-modal frameworks. This learning paradigm pulls positive pairs of samples closer while pushing negatives apart. In the uni-modal setting (e.g., image-based learning), previous research has shown that the strength of these forces can be controlled through the temperature parameter. In this work, we propose Multi-Modal Temperature and Margin Schedules (MM-TS), extending the concept of uni-modal temperature scheduling to multi-modal contrastive learning. Our method dynamically adjusts the temperature in the contrastive loss during training, modulating the attraction and repulsion forces in the multi-modal setting. Additionally, recognizing that standard multi-modal datasets often follow imbalanced, long-tail distributions, we adapt the temperature based on the local distribution of each training sample. Specifically, samples from dense clusters are assigned a higher temperature to better preserve their semantic structure. Furthermore, we demonstrate that temperature scheduling can be effectively integrated within a max-margin framework, thereby unifying the two predominant approaches in multi-modal contrastive learning: InfoNCE loss and max-margin objective. We evaluate our approach on four widely used image- and video-language datasets, Flickr30K, MSCOCO, EPIC-KITCHENS-100, and YouCook2, and show that our dynamic temperature and margin schedules improve performance and lead to new state-of-the-art results in the field.

Related papers

• TeMo: Temperature Modulation for Multimodal Contrastive Learning [69.50376336775496]
  Contrastive learning approaches achieve strong performance by training models to bring similar samples closer while pushing dissimilar samples apart.<n>We introduce TeMo, Temperature Modulation framework, a similarity-based modulation approach that adaptively adjusts the temperature for each positive-negative pair according to their similarity.<n>Our approach seamlessly integrates temperature-modulated multimodal and unimodal losses with the standard multimodal contrastive loss by gradually transitioning between them.
  arXiv  Detail & Related papers  (2026-09-07T14:23:02Z)
• Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RL [30.357975264905978]
  We propose a hierarchical reinforcement learning framework that learns to control sampling temperature during generation.<n>At each decoding step, the model selects a temperature based on its hidden state and samples the next token from the resulting distribution.<n>Temperature and token policies are jointly optimized from downstream rewards using a coordinate ascent scheme.
  arXiv  Detail & Related papers  (2026-02-13T15:42:59Z)
• Dynamic Temperature Scheduler for Knowledge Distillation [8.855130508913513]
  Knowledge Distillation (KD) trains a smaller student model using a large, pre-trained teacher model.<n>Traditional methods use a fixed temperature throughout training, which is suboptimal.<n>We introduce Dynamic Temperature Scheduler (DTS), which adjusts temperature dynamically based on the cross-entropy loss gap between teacher and student.
  arXiv  Detail & Related papers  (2025-11-14T16:03:22Z)
• LLM-Oriented Token-Adaptive Knowledge Distillation [64.08412563818662]
  We propose a novel framework that adapts the distillation process to the real-time learning state of each token.<n>AdaKD consists of two synergistic modules driven by a unified token difficulty metric.<n>As a plug-and-play framework, AdaKD can consistently improve the performance of various distillation methods on multiple model architectures and benchmarks.
  arXiv  Detail & Related papers  (2025-10-13T16:55:07Z)
• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• Continual Multimodal Contrastive Learning [99.53621521696051]
  Multimodal Contrastive Learning (MCL) advances in aligning different modalities and generating multimodal representations in a joint space.<n>However, a critical yet often overlooked challenge remains: multimodal data is rarely collected in a single process, and training from scratch is computationally expensive.<n>In this paper, we formulate CMCL through two specialized principles of stability and plasticity.<n>We theoretically derive a novel optimization-based method, which projects updated gradients from dual sides onto subspaces where any gradient is prevented from interfering with the previously learned knowledge.
  arXiv  Detail & Related papers  (2025-03-19T07:57:08Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training [58.20089993899729]
  This paper proposes TempBalance, a straightforward yet effective layerwise learning rate method. We show that TempBalance significantly outperforms ordinary SGD and carefully-tuned spectral norm regularization. We also show that TempBalance outperforms a number of state-of-the-art metrics and schedulers.
  arXiv  Detail & Related papers  (2023-12-01T05:38:17Z)
• Dynamically Scaled Temperature in Self-Supervised Contrastive Learning [11.133502139934437]
  We focus on improving the performance of InfoNCE loss in self-supervised learning by proposing a novel cosine similarity dependent temperature scaling function. Experimental evidence shows that the proposed framework outperforms the contrastive loss-based SSL algorithms.
  arXiv  Detail & Related papers  (2023-08-02T13:31:41Z)
• Tensor Network for Supervised Learning at Finite Temperature [1.4717465036484292]
  finite temperature tensor network (FTTN) imports thermal perturbation into matrix product states framework. FTTN regards it as thermal loss computed from entanglement with environment. Temperature-like parameter can be automatically optimized, which gives each database an individual temperature.
  arXiv  Detail & Related papers  (2021-04-09T05:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.