CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility

• URL: http://arxiv.org/abs/2603.12872v1
• Date: Fri, 13 Mar 2026 10:18:27 GMT
• Title: CLARIN-PT-LDB: An Open LLM Leaderboard for Portuguese to assess Language, Culture and Civility
• Abstract summary: This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT)<n>This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language.
• Score: 2.5205815072896134
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper reports on the development of a leaderboard of Open Large Language Models (LLM) for European Portuguese (PT-PT), and on its associated benchmarks. This leaderboard comes as a way to address a gap in the evaluation of LLM for European Portuguese, which so far had no leaderboard dedicated to this variant of the language. The paper also reports on novel benchmarks, including some that address aspects of performance that so far have not been available in benchmarks for European Portuguese, namely model safeguards and alignment to Portuguese culture. The leaderboard is available at https://huggingface.co/spaces/PORTULAN/portuguese-llm-leaderboard.

Related papers

• Beyond Multilingual Averages: MTEB-PT, a Benchmark for Portuguese Sentence Encoders [42.78591555984395]
  Portuguese is one of the most widely spoken languages in the world.<n> embedding models are often selected based on English or multilingual metrics, while their effectiveness in Portuguese remains unclear.<n>We present MTEB-PT, a Portuguese benchmark constructed from a subset of MMTEB.
  arXiv  Detail & Related papers  (2026-07-05T01:06:29Z)
• A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision-Language Models [50.00033296815133]
  Blind and low-vision (BLV) audiences remain underserved by visual art descriptions.<n>Pilot study investigates curator-guided multilingual art description with Qwen2.5-VL-3B-Instruct for German, Romanian, and Serbian.
  arXiv  Detail & Related papers  (2026-05-29T09:47:45Z)
• AMALIA Technical Report: A Fully Open Source Large Language Model for European Portuguese [12.322686257512428]
  We introduce AMALIA, a fully open LLM that prioritizes pt-PT by using more high-quality pt-PT data during both the mid- and post-training stages.<n>We release a suite of pt-PT benchmarks that includes translated standard tasks and four new datasets targeting pt-PT generation, linguistic competence, and pt-PT/pt-BR bias.<n>Experiments show that AMALIA matches strong baselines on translated benchmarks while substantially improving performance on pt-PT-specific evaluations.
  arXiv  Detail & Related papers  (2026-03-27T15:22:33Z)
• La Leaderboard: A Large Language Model Leaderboard for Spanish Varieties and Languages of Spain and Latin America [33.48097838499165]
  We present La Leaderboard, the first open-source leaderboard to evaluate generative Large Language Models.<n>This initial version combines 66 datasets in Basque, Catalan, Galician, and different Spanish varieties.<n>We explain our methodology, including guidance on selecting the most suitable evaluation setup for each downstream task.
  arXiv  Detail & Related papers  (2025-07-01T17:50:48Z)
• Enhancing Portuguese Variety Identification with Cross-Domain Approaches [2.31011809034817]
  We develop a cross-domain language variety identifier (LVI) to discriminate between European and Brazilian Portuguese.<n>Although this research focuses on two Portuguese varieties, our contribution can be extended to other varieties and languages.
  arXiv  Detail & Related papers  (2025-02-20T09:31:48Z)
• EuroLLM: Multilingual Language Models for Europe [76.89545643715368]
  We introduce the EuroLLM project, aimed at developing a suite of open-weight multilingual LLMs. We outline the progress made to date, detailing our data collection and filtering process. We report our performance on multilingual general benchmarks and machine translation.
  arXiv  Detail & Related papers  (2024-09-24T16:51:36Z)
• From Brazilian Portuguese to European Portuguese [2.048226951354646]
  Brazilian Portuguese and European Portuguese are two varieties of the same language. There is a significant disproportion in the availability of resources between the two variants. This inequity can impact the quality of translation services accessible to European Portuguese speakers.
  arXiv  Detail & Related papers  (2024-08-14T10:58:48Z)
• Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languages [48.40607157158246]
  Large Language Models (LLMs) perform better on high-resource languages like English, German, and French, while their capabilities in low-resource languages remain inadequate.<n>We propose the Language Ranker, an intrinsic metric designed to benchmark and rank languages based on LLM performance using internal representations.<n>Our analysis reveals that high-resource languages exhibit higher similarity scores with English, demonstrating superior performance, while low-resource languages show lower similarity scores.
  arXiv  Detail & Related papers  (2024-04-17T16:53:16Z)
• Introducing Bode: A Fine-Tuned Large Language Model for Portuguese Prompt-Based Task [1.158680734110387]
  This work proposes a fine-tuned LLaMA 2-based model for Portuguese prompts named Bode. We evaluate the performance of this model in classification tasks using the zero-shot approach with in-context learning.
  arXiv  Detail & Related papers  (2024-01-05T17:15:01Z)
• Extrapolating Large Language Models to Non-English by Aligning Languages [109.09051737966178]
  Existing large language models show disparate capability across different languages. In this paper, we empower pre-trained LLMs on non-English languages by building semantic alignment across languages.
  arXiv  Detail & Related papers  (2023-08-09T13:32:06Z)
• BLUEX: A benchmark based on Brazilian Leading Universities Entrance eXams [4.9069311006119865]
  We introduce BLUEX, a dataset of entrance exams from the two leading universities in Brazil: UNI CAMP and USP. The dataset includes annotated metadata for evaluating the performance of NLP models on a variety of subjects. We establish a benchmark through experiments with state-of-the-art LMs, demonstrating its potential for advancing the state-of-the-art in natural language understanding and reasoning in Portuguese.
  arXiv  Detail & Related papers  (2023-07-11T16:25:09Z)
• CUGE: A Chinese Language Understanding and Generation Evaluation Benchmark [144.05723617401674]
  General-purpose language intelligence evaluation has been a longstanding goal for natural language processing. We argue that for general-purpose language intelligence evaluation, the benchmark itself needs to be comprehensive and systematic. We propose CUGE, a Chinese Language Understanding and Generation Evaluation benchmark with the following features.
  arXiv  Detail & Related papers  (2021-12-27T11:08:58Z)
• Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings [72.69253034282035]
  We exploit a language independent multilingual sentence representation to easily generalize to a new language. Blindly decoding from Portuguese using a basesystem containing several Romance languages we achieve scores of 36.4 BLEU for Portuguese-English and 12.8 BLEU for Russian-English. We explore a more practical adaptation approach through non-iterative backtranslation, exploiting our model's ability to produce high quality translations.
  arXiv  Detail & Related papers  (2021-03-11T14:22:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.