MINERS: Multilingual Language Models as Semantic Retrievers

• URL: http://arxiv.org/abs/2406.07424v3
• Date: Tue, 24 Sep 2024 15:43:28 GMT
• Title: MINERS: Multilingual Language Models as Semantic Retrievers
• Authors: Genta Indra Winata, Ruochen Zhang, David Ifeoluwa Adelani,
• Abstract summary: This paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual language models in semantic retrieval tasks. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches.
• Score: 23.686762008696547
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Words have been represented in a high-dimensional vector space that encodes their semantic similarities, enabling downstream applications such as retrieving synonyms, antonyms, and relevant contexts. However, despite recent advances in multilingual language models (LMs), the effectiveness of these models' representations in semantic retrieval contexts has not been comprehensively explored. To fill this gap, this paper introduces the MINERS, a benchmark designed to evaluate the ability of multilingual LMs in semantic retrieval tasks, including bitext mining and classification via retrieval-augmented contexts. We create a comprehensive framework to assess the robustness of LMs in retrieving samples across over 200 diverse languages, including extremely low-resource languages in challenging cross-lingual and code-switching settings. Our results demonstrate that by solely retrieving semantically similar embeddings yields performance competitive with state-of-the-art approaches, without requiring any fine-tuning.

Related papers

• Tomato, Tomahto, Tomate: Measuring the Role of Shared Semantics among Subwords in Multilingual Language Models [88.07940818022468]
  We take an initial step on measuring the role of shared semantics among subwords in the encoder-only multilingual language models (mLMs) We form "semantic tokens" by merging the semantically similar subwords and their embeddings. inspections on the grouped subwords show that they exhibit a wide range of semantic similarities.
  arXiv  Detail & Related papers  (2024-11-07T08:38:32Z)
• Enhancing Cross-lingual Sentence Embedding for Low-resource Languages with Word Alignment [13.997006139875563]
  Cross-lingual word representation in low-resource languages is notably under-aligned with that in high-resource languages in current models. We introduce a novel framework that explicitly aligns words between English and eight low-resource languages, utilizing off-the-shelf word alignment models.
  arXiv  Detail & Related papers  (2024-04-03T05:58:53Z)
• Evaluation of Semantic Search and its Role in Retrieved-Augmented-Generation (RAG) for Arabic Language [0.0]
  This paper endeavors to establish a straightforward yet potent benchmark for semantic search in Arabic. To precisely evaluate the effectiveness of these metrics and the dataset, we conduct our assessment of semantic search within the framework of retrieval augmented generation (RAG)
  arXiv  Detail & Related papers  (2024-03-27T08:42:31Z)
• Semantic enrichment towards efficient speech representations [9.30840529284715]
  This study investigates a specific in-domain semantic enrichment of the SAMU-XLSR model. We show the benefits of the use of same-domain French and Italian benchmarks for low-resource language portability.
  arXiv  Detail & Related papers  (2023-07-03T19:52:56Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Retrofitting Multilingual Sentence Embeddings with Abstract Meaning Representation [70.58243648754507]
  We introduce a new method to improve existing multilingual sentence embeddings with Abstract Meaning Representation (AMR) Compared with the original textual input, AMR is a structured semantic representation that presents the core concepts and relations in a sentence explicitly and unambiguously. Experiment results show that retrofitting multilingual sentence embeddings with AMR leads to better state-of-the-art performance on both semantic similarity and transfer tasks.
  arXiv  Detail & Related papers  (2022-10-18T11:37:36Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• AStitchInLanguageModels: Dataset and Methods for the Exploration of Idiomaticity in Pre-Trained Language Models [7.386862225828819]
  This work presents a novel dataset of naturally occurring sentences containing MWEs manually classified into a fine-grained set of meanings. We use this dataset in two tasks designed to test i) a language model's ability to detect idiom usage, and ii) the effectiveness of a language model in generating representations of sentences containing idioms.
  arXiv  Detail & Related papers  (2021-09-09T16:53:17Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization [98.61159823343036]
  We present the Word-in-Context dataset (WiC) for assessing the ability to correctly model distinct meanings of a word. We put forward a large multilingual benchmark, XL-WiC, featuring gold standards in 12 new languages. Experimental results show that even when no tagged instances are available for a target language, models trained solely on the English data can attain competitive performance.
  arXiv  Detail & Related papers  (2020-10-13T15:32:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.