MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
- URL: http://arxiv.org/abs/2005.00052v3
- Date: Tue, 6 Oct 2020 10:17:45 GMT
- Title: MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
- Authors: Jonas Pfeiffer, Ivan Vuli\'c, Iryna Gurevych, Sebastian Ruder
- Abstract summary: We propose MAD-X, an adapter-based framework that enables high portability and parameter-efficient transfer to arbitrary tasks and languages.
MAD-X outperforms the state of the art in cross-lingual transfer across a representative set of typologically diverse languages on named entity recognition and causal commonsense reasoning.
- Score: 136.09386219006123
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The main goal behind state-of-the-art pre-trained multilingual models such as
multilingual BERT and XLM-R is enabling and bootstrapping NLP applications in
low-resource languages through zero-shot or few-shot cross-lingual transfer.
However, due to limited model capacity, their transfer performance is the
weakest exactly on such low-resource languages and languages unseen during
pre-training. We propose MAD-X, an adapter-based framework that enables high
portability and parameter-efficient transfer to arbitrary tasks and languages
by learning modular language and task representations. In addition, we
introduce a novel invertible adapter architecture and a strong baseline method
for adapting a pre-trained multilingual model to a new language. MAD-X
outperforms the state of the art in cross-lingual transfer across a
representative set of typologically diverse languages on named entity
recognition and causal commonsense reasoning, and achieves competitive results
on question answering. Our code and adapters are available at AdapterHub.ml
Related papers
- Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally.
Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
arXiv Detail & Related papers (2023-06-13T08:08:08Z) - Cross-Lingual Transfer with Target Language-Ready Task Adapters [66.5336029324059]
BAD-X, an extension of the MAD-X framework, achieves improved transfer at the cost of MAD-X's modularity.
We aim to take the best of both worlds by fine-tuning task adapters adapted to the target language.
arXiv Detail & Related papers (2023-06-05T10:46:33Z) - Language-Family Adapters for Low-Resource Multilingual Neural Machine
Translation [129.99918589405675]
Large multilingual models trained with self-supervision achieve state-of-the-art results in a wide range of natural language processing tasks.
Multilingual fine-tuning improves performance on low-resource languages but requires modifying the entire model and can be prohibitively expensive.
We propose training language-family adapters on top of mBART-50 to facilitate cross-lingual transfer.
arXiv Detail & Related papers (2022-09-30T05:02:42Z) - xGQA: Cross-Lingual Visual Question Answering [100.35229218735938]
xGQA is a new multilingual evaluation benchmark for the visual question answering task.
We extend the established English GQA dataset to 7 typologically diverse languages.
We propose new adapter-based approaches to adapt multimodal transformer-based models to become multilingual.
arXiv Detail & Related papers (2021-09-13T15:58:21Z) - UNKs Everywhere: Adapting Multilingual Language Models to New Scripts [103.79021395138423]
Massively multilingual language models such as multilingual BERT (mBERT) and XLM-R offer state-of-the-art cross-lingual transfer performance on a range of NLP tasks.
Due to their limited capacity and large differences in pretraining data, there is a profound performance gap between resource-rich and resource-poor target languages.
We propose novel data-efficient methods that enable quick and effective adaptation of pretrained multilingual models to such low-resource languages and unseen scripts.
arXiv Detail & Related papers (2020-12-31T11:37:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.