Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval

• URL: http://arxiv.org/abs/2505.19356v2
• Date: Tue, 10 Jun 2025 13:33:12 GMT
• Title: Optimized Text Embedding Models and Benchmarks for Amharic Passage Retrieval
• Authors: Kidist Amde Mekonnen, Yosef Worku Alemneh, Maarten de Rijke,
• Abstract summary: We introduce Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones.<n>Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10.<n>More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M) remain competitive while being over 13x smaller.
• Score: 49.1574468325115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural retrieval methods using transformer-based pre-trained language models have advanced multilingual and cross-lingual retrieval. However, their effectiveness for low-resource, morphologically rich languages such as Amharic remains underexplored due to data scarcity and suboptimal tokenization. We address this gap by introducing Amharic-specific dense retrieval models based on pre-trained Amharic BERT and RoBERTa backbones. Our proposed RoBERTa-Base-Amharic-Embed model (110M parameters) achieves a 17.6% relative improvement in MRR@10 and a 9.86% gain in Recall@10 over the strongest multilingual baseline, Arctic Embed 2.0 (568M parameters). More compact variants, such as RoBERTa-Medium-Amharic-Embed (42M), remain competitive while being over 13x smaller. Additionally, we train a ColBERT-based late interaction retrieval model that achieves the highest MRR@10 score (0.843) among all evaluated models. We benchmark our proposed models against both sparse and dense retrieval baselines to systematically assess retrieval effectiveness in Amharic. Our analysis highlights key challenges in low-resource settings and underscores the importance of language-specific adaptation. To foster future research in low-resource IR, we publicly release our dataset, codebase, and trained models at https://github.com/kidist-amde/amharic-ir-benchmarks.

Related papers

• Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples [58.55904048776596]
  Most open-source multilingual reward models are primarily trained on preference datasets in high-resource languages.<n>We propose RELIC, a novel in-context learning framework for reward modeling in low-resource Indic languages.
  arXiv  Detail & Related papers  (2025-06-19T17:56:16Z)
• Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language [4.276396344868335]
  We create resources to facilitate the adoption of Large Language Models (LLMs)<n>We collect the largest Macedonian corpus to date, consisting of 40GB of textual data and totaling 3.5B words.<n>We train domestic-yak, a state-of-the-art 8B- parameter model, on our curated datasets and evaluate it against eight baseline models.
  arXiv  Detail & Related papers  (2025-06-11T09:46:58Z)
• Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque [34.70526082204771]
  Instructing language models with user intent requires large instruction datasets, which are only available for a limited set of languages.<n>We assume a realistic scenario for low-resource languages, where only the following are available: corpora in the target language, existing open-weight multilingual base and instructed backbone LLMs, and synthetically generated instructions sampled from the instructed backbone.
  arXiv  Detail & Related papers  (2025-06-09T09:54:47Z)
• Subasa -- Adapting Language Models for Low-resourced Offensive Language Detection in Sinhala [9.298909305675595]
  We introduce four models: "Subasa-XLM-R", which incorporates an intermediate Pre-Finetuning step using Masked Rationale Prediction.<n>We evaluate our models on the SOLD benchmark dataset for Sinhala offensive language detection.
  arXiv  Detail & Related papers  (2025-04-02T23:46:49Z)
• Whispering in Amharic: Fine-tuning Whisper for Low-resource Language [3.2858851789879595]
  This work explores fine-tuning OpenAI's Whisper automatic speech recognition model for Amharic.<n>We fine-tune it using datasets like Mozilla Common Voice, FLEURS, and the BDU-speech dataset.<n>The best-performing model, Whispersmall-am, significantly improves when finetuned on a mix of existing FLEURS data and new, unseen Amharic datasets.
  arXiv  Detail & Related papers  (2025-03-24T09:39:41Z)
• JaColBERTv2.5: Optimising Multi-Vector Retrievers to Create State-of-the-Art Japanese Retrievers with Constrained Resources [0.0]
  We introduce a novel training recipe, resulting in the JaColBERTv2.5 model. With only 110 million parameters and trained in under 15 hours, JaColBERTv2.5 significantly outperforms all existing methods. To support future research, we make our final models, intermediate checkpoints and all data used publicly available.
  arXiv  Detail & Related papers  (2024-07-30T11:42:19Z)
• OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval [91.76575626229824]
  We present OneAligner, an alignment model specially designed for sentence retrieval tasks. When trained with all language pairs of a large-scale parallel multilingual corpus (OPUS-100), this model achieves the state-of-the-art result. We conclude through empirical results and analyses that the performance of the sentence alignment task depends mostly on the monolingual and parallel data size.
  arXiv  Detail & Related papers  (2022-05-17T19:52:42Z)
• RoBLEURT Submission for the WMT2021 Metrics Task [72.26898579202076]
  We present our submission to the Shared Metrics Task: RoBLEURT. Our model reaches state-of-the-art correlations with the WMT 2020 human annotations upon 8 out of 10 to-English language pairs.
  arXiv  Detail & Related papers  (2022-04-28T08:49:40Z)
• Learning Compact Metrics for MT [21.408684470261342]
  We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model. We show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT's performance using only a third of its parameters.
  arXiv  Detail & Related papers  (2021-10-12T20:39:35Z)
• Efficient Nearest Neighbor Language Models [114.40866461741795]
  Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore. We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
  arXiv  Detail & Related papers  (2021-09-09T12:32:28Z)
• AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages [75.08199398141744]
  We present AmericasNLI, an extension of XNLI (Conneau et al.), to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%.
  arXiv  Detail & Related papers  (2021-04-18T05:32:28Z)
• Improving Candidate Generation for Low-resource Cross-lingual Entity Linking [81.41804263432684]
  Cross-lingual entity linking (XEL) is the task of finding referents in a target-language knowledge base (KB) for mentions extracted from source-language texts. In this paper, we propose three improvements that (1) reduce the disconnect between entity mentions and KB entries, and (2) improve the robustness of the model to low-resource scenarios.
  arXiv  Detail & Related papers  (2020-03-03T05:32:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.