MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese

• URL: http://arxiv.org/abs/2607.04581v2
• Date: Tue, 07 Jul 2026 20:54:18 GMT
• Title: MTEB-BR: A Text Embedding Benchmark for Brazilian Portuguese
• Abstract summary: MTEB-BR is a benchmark of 22 native Brazilian-Portuguese tasks across seven categories.<n>We evaluate 93 models spanning 23M to 27B parameters.<n>We release every task, our code, and a public leaderboard, so practitioners can choose Portuguese embedding models on native evidence.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text embeddings for Portuguese have no dedicated benchmark: evaluation rests on translated corpora such as English MS MARCO or on thin multilingual coverage, with native tasks scattered and unconsolidated. We introduce MTEB-BR, a benchmark of 22 native Brazilian-Portuguese tasks across seven categories (classification, multilabel classification, pair classification, semantic textual similarity, clustering, retrieval, and reranking), admitting only data created or found in Portuguese and excluding translations by construction. We evaluate 93 models spanning 23M to 27B parameters: 73 open-weight and 20 closed commercial APIs. Alongside the leaderboard we report a statistical layer for every headline comparison: per-task bootstrap confidence intervals, paired-bootstrap significance, a task- and instance-level discrimination analysis (how sharply each task separates models) adapted from Item Response Theory, and a cross-leaderboard correlation. Three findings stand out. The benchmark cleanly separates about a dozen tiers of models, though the top six are statistically too close to order. An openly licensed, self-hostable model reaches that leading tier, so strong Portuguese embedding quality does not require a commercial API. And a model's rank on the global multilingual leaderboard predicts its Portuguese rank only moderately (Spearman rho = 0.75 over 55 shared models; one model ranks 3rd there and 49th here), so a native benchmark measures something the multilingual boards do not. We release every task, our code, and a public leaderboard, so practitioners can choose Portuguese embedding models on native evidence.

Related papers

• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders [42.78591555984395]
  Portuguese is one of the most widely spoken languages in the world.<n> embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear.<n>We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB.
  arXiv  Detail & Related papers  (2026-07-05T01:06:29Z)
• MARCA: A Checklist-Based Benchmark for Multilingual Web Search [8.678622777553263]
  We present textscMARCA, a benchmark for evaluating large language models (LLMs) on web-based information seeking.<n>We evaluate 14 models under two interaction settings: a Basic framework with direct web search and scraping, and an Orchestrator framework that enables task decomposition via delegated subagents.
  arXiv  Detail & Related papers  (2026-04-15T21:54:27Z)
• CAPITU: A Benchmark for Evaluating Instruction-Following in Brazilian Portuguese with Literary Context [8.678622777553267]
  CAPITU is a benchmark for evaluating instruction-following capabilities of Large Language Models (LLMs) in Brazilian Portuguese.<n>The benchmark comprises 59 instruction types organized into seven categories, all designed to be automatically verifiable.<n>We evaluate 18 state-of-the-art models across single-turn and multi-turn settings.
  arXiv  Detail & Related papers  (2026-03-23T21:16:54Z)
• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• CLaC at DISRPT 2025: Hierarchical Adapters for Cross-Framework Multi-lingual Discourse Relation Classification [0.0509780930114934]
  Task 3 introduces a unified set of 17 discourse relation labels across 39 corpora in 16 languages and six discourse frameworks.<n>We first benchmark the task by fine-tuning multilingual BERT-based models with two argument-ordering strategies and progressive unfreezing ratios.<n>We then evaluate prompt-based large language models in zero-shot and few-shot settings to understand how LLMs respond to the newly proposed unified labels.
  arXiv  Detail & Related papers  (2025-09-21T03:34:31Z)
• MMTEB: Massive Multilingual Text Embedding Benchmark [85.18187649328792]
  We introduce the Massive Multilingual Text Embedding Benchmark (MMTEB)<n>MMTEB covers over 500 quality-controlled evaluation tasks across 250+ languages.<n>We develop several highly multilingual benchmarks, which we use to evaluate a representative set of models.
  arXiv  Detail & Related papers  (2025-02-19T10:13:43Z)
• MTEB-French: Resources for French Sentence Embedding Evaluation and Analysis [1.5761916307614148]
  We propose the first benchmark of sentence embeddings for French. We compare 51 carefully selected embedding models on a large scale. We find that even if no model is the best on all tasks, large multilingual models pre-trained on sentence similarity perform exceptionally well.
  arXiv  Detail & Related papers  (2024-05-30T20:34:37Z)
• DeMuX: Data-efficient Multilingual Learning [57.37123046817781]
  DEMUX is a framework that prescribes exact data-points to label from vast amounts of unlabelled multilingual data. Our end-to-end framework is language-agnostic, accounts for model representations, and supports multilingual target configurations.
  arXiv  Detail & Related papers  (2023-11-10T20:09:08Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• OneAligner: Zero-shot Cross-lingual Transfer with One Rich-Resource Language Pair for Low-Resource Sentence Retrieval [91.76575626229824]
  We present OneAligner, an alignment model specially designed for sentence retrieval tasks. When trained with all language pairs of a large-scale parallel multilingual corpus (OPUS-100), this model achieves the state-of-the-art result. We conclude through empirical results and analyses that the performance of the sentence alignment task depends mostly on the monolingual and parallel data size.
  arXiv  Detail & Related papers  (2022-05-17T19:52:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.