Combining Contrastive and Non-Contrastive Losses for Fine-Tuning
Pretrained Models in Speech Analysis
- URL: http://arxiv.org/abs/2211.01964v1
- Date: Fri, 21 Oct 2022 19:58:37 GMT
- Title: Combining Contrastive and Non-Contrastive Losses for Fine-Tuning
Pretrained Models in Speech Analysis
- Authors: Florian Lux, Ching-Yi Chen, Ngoc Thang Vu
- Abstract summary: We propose a two step approach to finetuning paralinguistic properties.
First we improve the embedding space, then we train an adapter to bridge the gap from the embedding space to a classification task.
Our approach consistently outperforms baselines that are finetuned end-to-end on multiple tasks and surpasses a benchmark on state-of-the-art emotion classification.
- Score: 25.707717591185386
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Embedding paralinguistic properties is a challenging task as there are only a
few hours of training data available for domains such as emotional speech. One
solution to this problem is to pretrain a general self-supervised speech
representation model on large amounts of unlabeled speech. This pretrained
model is then finetuned to a specific task. Paralinguistic properties however
have notoriously high class variance, making the finetuning ineffective. In
this work, we propose a two step approach to this. First we improve the
embedding space, then we train an adapter to bridge the gap from the embedding
space to a classification task. In order to improve the class invariance we use
a combination of contrastive and non-contrastive losses to explicitly optimize
for class invariant, yet discriminative features. Our approach consistently
outperforms baselines that are finetuned end-to-end on multiple tasks and
surpasses a benchmark on state-of-the-art emotion classification.
Related papers
- Match me if you can: Semantic Correspondence Learning with Unpaired
Images [82.05105090432025]
We propose a simple yet effective method that performs training with unlabeled pairs to complement both limited image pairs and sparse point pairs.
Using a simple teacher-student framework, we offer reliable pseudo correspondences to the student network via machine supervision.
Our models outperform the milestone baselines, including state-of-the-art methods on semantic correspondence benchmarks.
arXiv Detail & Related papers (2023-11-30T13:22:15Z) - CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class
Classification [57.62886091828512]
We propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix) for many-class classification.
Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification.
arXiv Detail & Related papers (2022-11-11T03:45:59Z) - Hidden State Variability of Pretrained Language Models Can Guide
Computation Reduction for Transfer Learning [16.60284838029852]
We investigate whether one could make a task-specific selection on which subset of the layers to adapt.
We propose to select layers based on the variability of their hidden states given a task-specific corpus.
arXiv Detail & Related papers (2022-10-18T17:58:43Z) - Learning Debiased and Disentangled Representations for Semantic
Segmentation [52.35766945827972]
We propose a model-agnostic and training scheme for semantic segmentation.
By randomly eliminating certain class information in each training iteration, we effectively reduce feature dependencies among classes.
Models trained with our approach demonstrate strong results on multiple semantic segmentation benchmarks.
arXiv Detail & Related papers (2021-10-31T16:15:09Z) - Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
We propose textitPrototypical, which does not require fitting additional parameters given the embedding network.
Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced.
We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
arXiv Detail & Related papers (2021-10-22T01:55:01Z) - Not All Negatives are Equal: Label-Aware Contrastive Loss for
Fine-grained Text Classification [0.0]
We analyse the contrastive fine-tuning of pre-trained language models on two fine-grained text classification tasks.
We adaptively embed class relationships into a contrastive objective function to help differently weigh the positives and negatives.
We find that Label-aware Contrastive Loss outperforms previous contrastive methods.
arXiv Detail & Related papers (2021-09-12T04:19:17Z) - Robust Transfer Learning with Pretrained Language Models through
Adapters [40.45102278979193]
Transfer learning with large pretrained language models like BERT has become a dominating approach for most NLP tasks.
We propose a simple yet effective adapter-based approach to mitigate these issues.
Our experiments demonstrate that such a training scheme leads to improved stability and adversarial robustness in transfer learning to various downstream tasks.
arXiv Detail & Related papers (2021-08-05T02:30:13Z) - A Differentiable Language Model Adversarial Attack on Text Classifiers [10.658675415759697]
We propose a new black-box sentence-level attack for natural language processing.
Our method fine-tunes a pre-trained language model to generate adversarial examples.
We show that the proposed attack outperforms competitors on a diverse set of NLP problems for both computed metrics and human evaluation.
arXiv Detail & Related papers (2021-07-23T14:43:13Z) - Differentiable Language Model Adversarial Attacks on Categorical
Sequence Classifiers [0.0]
An adversarial attack paradigm explores various scenarios for the vulnerability of deep learning models.
We use a fine-tuning of a language model for adversarial attacks as a generator of adversarial examples.
Our model works for diverse datasets on bank transactions, electronic health records, and NLP datasets.
arXiv Detail & Related papers (2020-06-19T11:25:36Z) - TAVAT: Token-Aware Virtual Adversarial Training for Language
Understanding [55.16953347580948]
Gradient-based adversarial training is widely used in improving the robustness of neural networks.
It cannot be easily adapted to natural language processing tasks since the embedding space is discrete.
We propose a Token-Aware Virtual Adrial Training method to craft fine-grained perturbations.
arXiv Detail & Related papers (2020-04-30T02:03:24Z) - Pre-training Is (Almost) All You Need: An Application to Commonsense
Reasoning [61.32992639292889]
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks.
We introduce a new scoring method that casts a plausibility ranking task in a full-text format.
We show that our method provides a much more stable training phase across random restarts.
arXiv Detail & Related papers (2020-04-29T10:54:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.