Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised
Pre-Training
- URL: http://arxiv.org/abs/2104.01027v1
- Date: Fri, 2 Apr 2021 12:53:15 GMT
- Title: Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised
Pre-Training
- Authors: Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko,
Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel
Synnaeve, Michael Auli
- Abstract summary: We show that using target domain data during pre-training leads to large performance improvements across a variety of setups.
We find that pre-training on multiple domains improves performance generalization on domains not seen during training.
- Score: 67.71228426496013
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Self-supervised learning of speech representations has been a very active
research area but most work is focused on a single domain such as read audio
books for which there exist large quantities of labeled and unlabeled data. In
this paper, we explore more general setups where the domain of the unlabeled
data for pre-training data differs from the domain of the labeled data for
fine-tuning, which in turn may differ from the test data domain. Our
experiments show that using target domain data during pre-training leads to
large performance improvements across a variety of setups. On a large-scale
competitive setup, we show that pre-training on unlabeled in-domain data
reduces the gap between models trained on in-domain and out-of-domain labeled
data by 66%-73%. This has obvious practical implications since it is much
easier to obtain unlabeled target domain data than labeled data. Moreover, we
find that pre-training on multiple domains improves generalization performance
on domains not seen during training. Code and models will be made available at
https://github.com/pytorch/fairseq.
Related papers
- Task Oriented In-Domain Data Augmentation [38.525017729123114]
Large Language Models (LLMs) have shown superior performance in various applications and fields.
To achieve better performance on specialized domains such as law and advertisement, LLMs are often continue pre-trained on in-domain data.
We propose TRAIT, a task-oriented in-domain data augmentation framework.
arXiv Detail & Related papers (2024-06-24T14:58:11Z) - Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised
Domain Adaptation [88.5448806952394]
We consider unsupervised domain adaptation (UDA), where labeled data from a source domain and unlabeled data from a target domain are used to learn a classifier for the target domain.
We show that contrastive pre-training, which learns features on unlabeled source and target data and then fine-tunes on labeled source data, is competitive with strong UDA methods.
arXiv Detail & Related papers (2022-04-01T16:56:26Z) - A Survey of Unsupervised Domain Adaptation for Visual Recognition [2.8935588665357077]
Domain Adaptation (DA) aims to mitigate the domain shift problem when transferring knowledge from one domain to another.
Unsupervised DA (UDA) deals with a labeled source domain and an unlabeled target domain.
arXiv Detail & Related papers (2021-12-13T15:55:23Z) - Cross-domain Contrastive Learning for Unsupervised Domain Adaptation [108.63914324182984]
Unsupervised domain adaptation (UDA) aims to transfer knowledge learned from a fully-labeled source domain to a different unlabeled target domain.
We build upon contrastive self-supervised learning to align features so as to reduce the domain discrepancy between training and testing sets.
arXiv Detail & Related papers (2021-06-10T06:32:30Z) - Prototypical Cross-domain Self-supervised Learning for Few-shot
Unsupervised Domain Adaptation [91.58443042554903]
We propose an end-to-end Prototypical Cross-domain Self-Supervised Learning (PCS) framework for Few-shot Unsupervised Domain Adaptation (FUDA)
PCS not only performs cross-domain low-level feature alignment, but it also encodes and aligns semantic structures in the shared embedding space across domains.
Compared with state-of-the-art methods, PCS improves the mean classification accuracy over different domain pairs on FUDA by 10.5%, 3.5%, 9.0%, and 13.2% on Office, Office-Home, VisDA-2017, and DomainNet, respectively.
arXiv Detail & Related papers (2021-03-31T02:07:42Z) - Domain Generalized Person Re-Identification via Cross-Domain Episodic
Learning [31.17248105464821]
We present an episodic learning scheme which advances meta learning strategies to exploit the observed source-domain labeled data.
Our experiments on four benchmark datasets confirm the superiority of our method over the state-of-the-arts.
arXiv Detail & Related papers (2020-10-19T14:42:29Z) - Improving Adversarial Robustness via Unlabeled Out-of-Domain Data [30.58040078862511]
We investigate how adversarial robustness can be enhanced by leveraging out-of-domain unlabeled data.
We show settings where we achieve better adversarial robustness when the unlabeled data come from a shifted domain rather than the same domain as the labeled data.
arXiv Detail & Related papers (2020-06-15T15:25:56Z) - Deep Domain-Adversarial Image Generation for Domain Generalisation [115.21519842245752]
Machine learning models typically suffer from the domain shift problem when trained on a source dataset and evaluated on a target dataset of different distribution.
To overcome this problem, domain generalisation (DG) methods aim to leverage data from multiple source domains so that a trained model can generalise to unseen domains.
We propose a novel DG approach based on emphDeep Domain-Adversarial Image Generation (DDAIG)
arXiv Detail & Related papers (2020-03-12T23:17:47Z) - Mind the Gap: Enlarging the Domain Gap in Open Set Domain Adaptation [65.38975706997088]
Open set domain adaptation (OSDA) assumes the presence of unknown classes in the target domain.
We show that existing state-of-the-art methods suffer a considerable performance drop in the presence of larger domain gaps.
We propose a novel framework to specifically address the larger domain gaps.
arXiv Detail & Related papers (2020-03-08T14:20:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.