The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design

• URL: http://arxiv.org/abs/2408.12503v1
• Date: Thu, 22 Aug 2024 15:53:23 GMT
• Title: The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design
• Authors: Artem Snegirev, Maria Tikhonova, Anna Maksimova, Alena Fenogenova, Alexander Abramov,
• Abstract summary: This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark.
• Score: 39.80182519545138
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embedding models play a crucial role in Natural Language Processing (NLP) by creating text embeddings used in various tasks such as information retrieval and assessing semantic text similarity. This paper focuses on research related to embedding models in the Russian language. It introduces a new Russian-focused embedding model called ru-en-RoSBERTa and the ruMTEB benchmark, the Russian version extending the Massive Text Embedding Benchmark (MTEB). Our benchmark includes seven categories of tasks, such as semantic textual similarity, text classification, reranking, and retrieval. The research also assesses a representative set of Russian and multilingual models on the proposed benchmark. The findings indicate that the new model achieves results that are on par with state-of-the-art models in Russian. We release the model ru-en-RoSBERTa, and the ruMTEB framework comes with open-source code, integration into the original framework and a public leaderboard.

Related papers

• Enhancing Modern Supervised Word Sense Disambiguation Models by Semantic Lexical Resources [11.257738983764499]
  Supervised models for Word Sense Disambiguation (WSD) currently yield to state-of-the-art results in the most popular benchmarks. We enhance "modern" supervised WSD models exploiting two popular SLRs: WordNet and WordNet Domains. We study the effect of different types of semantic features, investigating their interaction with local contexts encoded by means of mixtures of Word Embeddings or Recurrent Neural Networks.
  arXiv  Detail & Related papers  (2024-02-20T13:47:51Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• Entity-Assisted Language Models for Identifying Check-worthy Sentences [23.792877053142636]
  We propose a new uniform framework for text classification and ranking. Our framework combines the semantic analysis of the sentences, with additional entity embeddings obtained through the identified entities within the sentences. We extensively evaluate the effectiveness of our framework using two publicly available datasets from the CLEF's 2019 & 2020 CheckThat! Labs.
  arXiv  Detail & Related papers  (2022-11-19T12:03:30Z)
• FRMT: A Benchmark for Few-Shot Region-Aware Machine Translation [64.9546787488337]
  We present FRMT, a new dataset and evaluation benchmark for Few-shot Region-aware Machine Translation. The dataset consists of professional translations from English into two regional variants each of Portuguese and Mandarin Chinese.
  arXiv  Detail & Related papers  (2022-10-01T05:02:04Z)
• RuBioRoBERTa: a pre-trained biomedical language model for Russian language biomedical text mining [117.56261821197741]
  We present several BERT-based models for Russian language biomedical text mining. The models are pre-trained on a corpus of freely available texts in the Russian biomedical domain.
  arXiv  Detail & Related papers  (2022-04-08T09:18:59Z)
• Russian SuperGLUE 1.1: Revising the Lessons not Learned by Russian NLP models [53.95094814056337]
  This paper presents Russian SuperGLUE 1.1, an updated benchmark styled after GLUE for Russian NLP models. The new version includes a number of technical, user experience and methodological improvements. We provide the integration of Russian SuperGLUE with a framework for industrial evaluation of the open-source models, MOROCCO.
  arXiv  Detail & Related papers  (2022-02-15T23:45:30Z)
• RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark [5.258267224004844]
  We introduce an advanced Russian general language understanding evaluation benchmark -- RussianGLUE. For the first time, a benchmark of nine tasks, collected and organized analogically to the SuperGLUE methodology, was developed from scratch for the Russian language.
  arXiv  Detail & Related papers  (2020-10-29T20:31:39Z)
• Dataset for Automatic Summarization of Russian News [0.0]
  We present Gazeta, the first dataset for summarization of Russian news. We demonstrate that the dataset is a valid task for methods of text summarization for Russian.
  arXiv  Detail & Related papers  (2020-06-19T10:44:06Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.