Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker Representations

• URL: http://arxiv.org/abs/2404.14913v1
• Date: Tue, 23 Apr 2024 10:56:58 GMT
• Title: Additive Margin in Contrastive Self-Supervised Frameworks to Learn Discriminative Speaker Representations
• Authors: Theo Lepage, Reda Dehak,
• Abstract summary: We discuss the importance of Additive Margin (AM) in SimCLR and MoCo SSL methods to further separate positive from negative pairs. Implementing these two modifications to SimCLR improves performance and results in 7.85% EER on VoxCeleb1-O, outperforming other equivalent methods.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-Supervised Learning (SSL) frameworks became the standard for learning robust class representations by benefiting from large unlabeled datasets. For Speaker Verification (SV), most SSL systems rely on contrastive-based loss functions. We explore different ways to improve the performance of these techniques by revisiting the NT-Xent contrastive loss. Our main contribution is the definition of the NT-Xent-AM loss and the study of the importance of Additive Margin (AM) in SimCLR and MoCo SSL methods to further separate positive from negative pairs. Despite class collisions, we show that AM enhances the compactness of same-speaker embeddings and reduces the number of false negatives and false positives on SV. Additionally, we demonstrate the effectiveness of the symmetric contrastive loss, which provides more supervision for the SSL task. Implementing these two modifications to SimCLR improves performance and results in 7.85% EER on VoxCeleb1-O, outperforming other equivalent methods.

Related papers

• On the Effectiveness of Supervision in Asymmetric Non-Contrastive Learning [5.123232962822044]
  asymmetric non-contrastive learning (ANCL) often outperforms its contrastive learning counterpart in self-supervised representation learning. We study ANCL for supervised representation learning, coined SupSiam and SupBYOL, leveraging labels in ANCL to achieve better representations. Our analysis reveals that providing supervision to ANCL reduces intra-class variance, and the contribution of supervision should be adjusted to achieve the best performance.
  arXiv  Detail & Related papers  (2024-06-16T06:43:15Z)
• Towards Supervised Performance on Speaker Verification with Self-Supervised Learning by Leveraging Large-Scale ASR Models [0.0]
  Speech representations from large-scale ASR models contain valuable speaker information. We propose a framework to learn speaker representations in an SSL context by fine-tuning a pre-trained WavLM with a supervised loss. Our method achieves 0.99% EER on VoxCeleb1-O, establishing the new state-of-the-art on self-supervised SV.
  arXiv  Detail & Related papers  (2024-06-04T12:58:19Z)
• Towards Better Understanding of Contrastive Sentence Representation Learning: A Unified Paradigm for Gradient [20.37803751979975]
  Sentence Representation Learning (SRL) is a crucial task in Natural Language Processing (NLP) Many studies have investigated the similarities between contrastive and non-contrastive Self-Supervised Learning (SSL) But in ranking tasks (i.e., Semantic Textual Similarity (STS) in SRL), contrastive SSL significantly outperforms non-contrastive SSL.
  arXiv  Detail & Related papers  (2024-02-28T12:17:40Z)
• Relaxed Contrastive Learning for Federated Learning [48.96253206661268]
  We propose a novel contrastive learning framework to address the challenges of data heterogeneity in federated learning. Our framework outperforms all existing federated learning approaches by huge margins on the standard benchmarks.
  arXiv  Detail & Related papers  (2024-01-10T04:55:24Z)
• Improving Contrastive Learning of Sentence Embeddings with Focal-InfoNCE [13.494159547236425]
  This study introduces an unsupervised contrastive learning framework that combines SimCSE with hard negative mining. The proposed focal-InfoNCE function introduces self-paced modulation terms in the contrastive objective, downweighting the loss associated with easy negatives and encouraging the model focusing on hard negatives.
  arXiv  Detail & Related papers  (2023-10-10T18:15:24Z)
• Identical and Fraternal Twins: Fine-Grained Semantic Contrastive Learning of Sentence Representations [6.265789210037749]
  We introduce a novel Identical and Fraternal Twins of Contrastive Learning framework, capable of simultaneously adapting to various positive pairs generated by different augmentation techniques. We also present proof-of-concept experiments combined with the contrastive objective to prove the validity of the proposed Twins Loss.
  arXiv  Detail & Related papers  (2023-07-20T15:02:42Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• On Higher Adversarial Susceptibility of Contrastive Self-Supervised Learning [104.00264962878956]
  Contrastive self-supervised learning (CSL) has managed to match or surpass the performance of supervised learning in image and video classification. It is still largely unknown if the nature of the representation induced by the two learning paradigms is similar. We identify the uniform distribution of data representation over a unit hypersphere in the CSL representation space as the key contributor to this phenomenon. We devise strategies that are simple, yet effective in improving model robustness with CSL training.
  arXiv  Detail & Related papers  (2022-07-22T03:49:50Z)
• Adversarial Contrastive Learning via Asymmetric InfoNCE [64.42740292752069]
  We propose to treat adversarial samples unequally when contrasted with an asymmetric InfoNCE objective. In the asymmetric fashion, the adverse impacts of conflicting objectives between CL and adversarial learning can be effectively mitigated. Experiments show that our approach consistently outperforms existing Adversarial CL methods.
  arXiv  Detail & Related papers  (2022-07-18T04:14:36Z)
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
  Self-supervised learning has achieved great success in learning visual representations without data annotations. We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances. Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2021-07-20T06:53:07Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.