Selective Contrastive Learning For Gloss Free Sign Language Translation

• URL: http://arxiv.org/abs/2604.22374v1
• Date: Fri, 24 Apr 2026 09:08:45 GMT
• Title: Selective Contrastive Learning For Gloss Free Sign Language Translation
• Abstract summary: Sign language translation (SLT) converts continuous sign videos into spoken-language text.<n>Recent SLT systems adopt CLIP-like Vision-Language pretraining for cross-modal alignment.<n>But random in-batch contrast provides few, batch-dependent negatives and may mislabel semantically similar (or even identical) pairs as negatives.<n>We propose Selective Contrastive Learning for SLT (SCL-SLT) with a Pair Selection strategy.
• Score: 28.071457836616506
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sign language translation (SLT) converts continuous sign videos into spoken-language text, yet it remains challenging due to the intrinsic modality mismatch between visual signs and written text, particularly in gloss-free settings. Recent SLT systems increasingly adopt CLIP-like Vision-Language pretraining (VLP) for cross-modal alignment, but the random in-batch contrast provides few, batch-dependent negatives and may mislabel semantically similar (or even identical) pairs as negatives, introducing noisy and potentially inconsistent alignment supervision. In this work, we first conduct a preliminary trajectory-based analysis that tracks negative video-text similarity over training. The results show that only a small subset of negatives exhibits the desired behavior of being consistently pushed away, while the remaining negatives display heterogeneous and often non-decreasing similarity dynamics, suggesting that random in-batch negatives are frequently uninformative for effective alignment. Inspired by this, we propose Selective Contrastive Learning for SLT (SCL-SLT) with a Pair Selection (PS) strategy. PS scores candidate negatives using similarity dynamics from reference checkpoints and constructs mini-batches via a curriculum that progressively emphasizes more challenging negatives, thereby strengthening contrastive supervision while reducing the influence of noisy or semantically invalid negatives.

Related papers

• HNCSE: Advancing Sentence Embeddings via Hybrid Contrastive Learning with Hard Negatives [17.654412302780557]
  HNCSE is a novel contrastive learning framework that extends the leading SimCSE approach. The hallmark of HNCSE is its innovative use of hard negative samples to enhance the learning of both positive and negative samples.
  arXiv  Detail & Related papers  (2024-11-19T01:26:20Z)
• Enhancing Sound Source Localization via False Negative Elimination [58.87973081084927]
  Sound source localization aims to localize objects emitting the sound in visual scenes. Recent works obtaining impressive results typically rely on contrastive learning. We propose a novel audio-visual learning framework which is instantiated with two individual learning schemes.
  arXiv  Detail & Related papers  (2024-08-29T11:24:51Z)
• Contrastive Learning with Negative Sampling Correction [52.990001829393506]
  We propose a novel contrastive learning method named Positive-Unlabeled Contrastive Learning (PUCL) PUCL treats the generated negative samples as unlabeled samples and uses information from positive samples to correct bias in contrastive loss. PUCL can be applied to general contrastive learning problems and outperforms state-of-the-art methods on various image and graph classification tasks.
  arXiv  Detail & Related papers  (2024-01-13T11:18:18Z)
• Clustering-Aware Negative Sampling for Unsupervised Sentence Representation [24.15096466098421]
  ClusterNS is a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified K-means clustering algorithm to supply hard negatives and recognize in-batch false negatives during training.
  arXiv  Detail & Related papers  (2023-05-17T02:06:47Z)
• Adversarial Contrastive Learning via Asymmetric InfoNCE [64.42740292752069]
  We propose to treat adversarial samples unequally when contrasted with an asymmetric InfoNCE objective. In the asymmetric fashion, the adverse impacts of conflicting objectives between CL and adversarial learning can be effectively mitigated. Experiments show that our approach consistently outperforms existing Adversarial CL methods.
  arXiv  Detail & Related papers  (2022-07-18T04:14:36Z)
• Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negatives [17.90820242798732]
  Unsupervised contrastive learning methods still lag far behind the supervised counterparts. We propose switch-case augmentation to flip the case of the first letter of randomly selected words in a sentence. For negative samples, we sample hard negatives from the whole dataset based on a pre-trained language model.
  arXiv  Detail & Related papers  (2022-06-06T09:46:12Z)
• Debiased Contrastive Learning of Unsupervised Sentence Representations [88.58117410398759]
  Contrastive learning is effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. Previous works mostly adopt in-batch negatives or sample from training data at random. We present a new framework textbfDCLR to alleviate the influence of these improper negatives.
  arXiv  Detail & Related papers  (2022-05-02T05:07:43Z)
• PCL: Peer-Contrastive Learning with Diverse Augmentations for Unsupervised Sentence Embeddings [69.87899694963251]
  We propose a novel Peer-Contrastive Learning (PCL) with diverse augmentations. PCL constructs diverse contrastive positives and negatives at the group level for unsupervised sentence embeddings. PCL can perform peer-positive contrast as well as peer-network cooperation, which offers an inherent anti-bias ability.
  arXiv  Detail & Related papers  (2022-01-28T13:02:41Z)
• SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples [36.08601841321196]
  We propose contrastive learning for unsupervised sentence embedding with soft negative samples. We show that SNCSE can obtain state-of-the-art performance on semantic textual similarity task.
  arXiv  Detail & Related papers  (2022-01-16T06:15:43Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)
• SimCSE: Simple Contrastive Learning of Sentence Embeddings [10.33373737281907]
  This paper presents SimCSE, a contrastive learning framework for embeddings. We first describe an unsupervised approach, which takes an input sentence and predicts itself in a contrastive objective. We then incorporate annotated pairs from NLI datasets into contrastive learning by using "entailment" pairs as positives and "contradiction" pairs as hard negatives.
  arXiv  Detail & Related papers  (2021-04-18T11:27:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.