Distillation for Multilingual Information Retrieval

• URL: http://arxiv.org/abs/2405.00977v1
• Date: Thu, 2 May 2024 03:30:03 GMT
• Title: Distillation for Multilingual Information Retrieval
• Authors: Eugene Yang, Dawn Lawrie, James Mayfield,
• Abstract summary: Translate-Distill framework trains a cross-language neural dual-encoder model using translation and distillation. This work extends Translate-Distill and propose Translate-Distill (MTD) for Multilingual information retrieval. We show that ColBERT-X models trained with MTD outperform their counterparts trained ith Multilingual Translate-Train, by 5% to 25% in nDCG@20 and 15% to 45% in MAP.
• Score: 10.223578525761617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work in cross-language information retrieval (CLIR), where queries and documents are in different languages, has shown the benefit of the Translate-Distill framework that trains a cross-language neural dual-encoder model using translation and distillation. However, Translate-Distill only supports a single document language. Multilingual information retrieval (MLIR), which ranks a multilingual document collection, is harder to train than CLIR because the model must assign comparable relevance scores to documents in different languages. This work extends Translate-Distill and propose Multilingual Translate-Distill (MTD) for MLIR. We show that ColBERT-X models trained with MTD outperform their counterparts trained ith Multilingual Translate-Train, which is the previous state-of-the-art training approach, by 5% to 25% in nDCG@20 and 15% to 45% in MAP. We also show that the model is robust to the way languages are mixed in training batches. Our implementation is available on GitHub.

Related papers

• Translate-Distill: Learning Cross-Language Dense Retrieval by Translation and Distillation [17.211592060717713]
  This paper proposes Translate-Distill, in which knowledge distillation from either a monolingual cross-encoder or a CLIR cross-encoder is used to train a dual-encoder CLIR student model. This richer design space enables the teacher model to perform inference in an optimized setting, while training the student model directly for CLIR.
  arXiv  Detail & Related papers  (2024-01-09T20:40:49Z)
• Distilling Efficient Language-Specific Models for Cross-Lingual Transfer [75.32131584449786]
  Massively multilingual Transformers (MMTs) are widely used for cross-lingual transfer learning. MMTs' language coverage makes them unnecessarily expensive to deploy in terms of model size, inference time, energy, and hardware cost. We propose to extract compressed, language-specific models from MMTs which retain the capacity of the original MMTs for cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-06-02T17:31:52Z)
• Soft Prompt Decoding for Multilingual Dense Retrieval [30.766917713997355]
  We show that applying state-of-the-art approaches developed for cross-lingual information retrieval to MLIR tasks leads to sub-optimal performance. This is due to the heterogeneous and imbalanced nature of multilingual collections. We present KD-SPD, a novel soft prompt decoding approach for MLIR that implicitly "translates" the representation of documents in different languages into the same embedding space.
  arXiv  Detail & Related papers  (2023-05-15T21:17:17Z)
• Multilingual Multimodal Learning with Machine Translated Text [27.7207234512674]
  We investigate whether machine translating English multimodal data can be an effective proxy for the lack of readily available multilingual data. We propose two metrics for automatically removing such translations from the resulting datasets. In experiments on five tasks across 20 languages in the IGLUE benchmark, we show that translated data can provide a useful signal for multilingual multimodal learning.
  arXiv  Detail & Related papers  (2022-10-24T11:41:20Z)
• Multilingual ColBERT-X [11.768656900939048]
  ColBERT-X is a dense retrieval model for Cross Language Information Retrieval ( CLIR) In CLIR, documents are written in one natural language, while the queries are expressed in another. A related task is multilingual IR (MLIR) where the system creates a single ranked list of documents written in many languages.
  arXiv  Detail & Related papers  (2022-09-03T06:02:52Z)
• Breaking Down Multilingual Machine Translation [74.24795388967907]
  We show that multilingual training is beneficial to encoders in general, while it only benefits decoders for low-resource languages (LRLs) Our many-to-one models for high-resource languages and one-to-many models for LRLs outperform the best results reported by Aharoni et al.
  arXiv  Detail & Related papers  (2021-10-15T14:57:12Z)
• Cross-lingual Intermediate Fine-tuning improves Dialogue State Tracking [84.50302759362698]
  We enhance the transfer learning process by intermediate fine-tuning of pretrained multilingual models. We use parallel and conversational movie subtitles datasets to design cross-lingual intermediate tasks. We achieve impressive improvements (> 20% on goal accuracy) on the parallel MultiWoZ dataset and Multilingual WoZ dataset.
  arXiv  Detail & Related papers  (2021-09-28T11:22:38Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)
• Beyond English-Centric Multilingual Machine Translation [74.21727842163068]
  We create a true Many-to-Many multilingual translation model that can translate directly between any pair of 100 languages. We build and open source a training dataset that covers thousands of language directions with supervised data, created through large-scale mining. Our focus on non-English-Centric models brings gains of more than 10 BLEU when directly translating between non-English directions while performing competitively to the best single systems of WMT.
  arXiv  Detail & Related papers  (2020-10-21T17:01:23Z)
• Multilingual Translation with Extensible Multilingual Pretraining and Finetuning [77.33262578776291]
  Previous work has demonstrated that machine translation systems can be created by finetuning on bitext. We show that multilingual translation models can be created through multilingual finetuning. We demonstrate that pretrained models can be extended to incorporate additional languages without loss of performance.
  arXiv  Detail & Related papers  (2020-08-02T05:36:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.