Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark

• URL: http://arxiv.org/abs/2604.12744v1
• Date: Tue, 14 Apr 2026 13:57:29 GMT
• Title: Universal NER v2: Towards a Massively Multilingual Named Entity Recognition Benchmark
• Abstract summary: The Universal NER project is dedicated to building gold-standard multilingual Named Entity Recognition benchmark datasets.<n>The project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans.
• Score: 19.16689552291377
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While multilingual language models promise to bring the benefits of LLMs to speakers of many languages, gold-standard evaluation benchmarks in most languages to interrogate these assumptions remain scarce. The Universal NER project, now entering its fourth year, is dedicated to building gold-standard multilingual Named Entity Recognition (NER) benchmark datasets. Inspired by existing massively multilingual efforts for other core NLP tasks (e.g., Universal Dependencies), the project uses a general tagset and thorough annotation guidelines to collect standardized, cross-lingual annotations of named entity spans. The first installment (UNER v1) was released in 2024, and the project has continued and expanded since then, with various organizers, annotators, and collaborators in an active community.

Related papers

• LUSIFER: Language Universal Space Integration for Enhanced Multilingual Embeddings with Large Language Models [89.13128402847943]
  We present LUSIFER, a novel zero-shot approach that adapts LLM-based embedding models for multilingual tasks without requiring multilingual supervision.<n>LUSIFER's architecture combines a multilingual encoder, serving as a language-universal learner, with an LLM-based embedding model optimized for embedding-specific tasks.<n>We introduce a new benchmark encompassing 5 primary embedding tasks, 123 diverse datasets, and coverage across 14 languages.
  arXiv  Detail & Related papers  (2025-01-01T15:43:07Z)
• M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation [39.6123499117046]
  We propose a massively multilingual repository-level code completion benchmark covering 18 programming languages. Two types of fine-grained annotations (i.e., bucket-level and semantic-level) are provided on different completion scenarios. We also curate a massively multilingual instruction corpora M2RC- INSTRUCT dataset to improve the repository-level code completion abilities of existing code Large Language Models.
  arXiv  Detail & Related papers  (2024-10-28T15:58:41Z)
• OMGEval: An Open Multilingual Generative Evaluation Benchmark for Large Language Models [59.54423478596468]
  We introduce OMGEval, the first Open-source Multilingual Generative test set that can assess the capability of LLMs in different languages. For each language, OMGEval provides 804 open-ended questions, covering a wide range of important capabilities of LLMs. Specifically, the current version of OMGEval includes 5 languages (i.e., Zh, Ru, Fr, Es, Ar)
  arXiv  Detail & Related papers  (2024-02-21T04:42:41Z)
• Universal NER: A Gold-Standard Multilingual Named Entity Recognition Benchmark [39.01204607174688]
  We introduce Universal NER (UNER), an open, community-driven project to develop gold-standard NER benchmarks in many languages. UNER v1 contains 18 datasets annotated with named entities in a cross-lingual consistent schema across 12 diverse languages.
  arXiv  Detail & Related papers  (2023-11-15T17:09:54Z)
• IXA/Cogcomp at SemEval-2023 Task 2: Context-enriched Multilingual Named Entity Recognition using Knowledge Bases [53.054598423181844]
  We present a novel NER cascade approach comprising three steps. We empirically demonstrate the significance of external knowledge bases in accurately classifying fine-grained and emerging entities. Our system exhibits robust performance in the MultiCoNER2 shared task, even in the low-resource language setting.
  arXiv  Detail & Related papers  (2023-04-20T20:30:34Z)
• CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation [113.99145386490639]
  Cross-lingual NER can transfer knowledge between languages via aligned cross-lingual representations or machine translation results. We propose a Cross-lingual Entity Projection framework (CROP) to enable zero-shot cross-lingual NER. We adopt a multilingual labeled sequence translation model to project the tagged sequence back to the target language and label the target raw sentence.
  arXiv  Detail & Related papers  (2022-10-13T13:32:36Z)
• UNER: Universal Named-Entity RecognitionFramework [0.0]
  We create the first multilingual UNER corpus: the SETimesparallel corpus annotated for named-entities. The English SETimescorpus will be annotated using existing tools and knowledge bases. The resulting annotations will be propagated automatically to other languages within the SE-Times corpora.
  arXiv  Detail & Related papers  (2020-10-23T13:53:31Z)
• GLUECoS : An Evaluation Benchmark for Code-Switched NLP [17.066725832825423]
  We present an evaluation benchmark, GLUECoS, for code-switched languages. We present results on several NLP tasks in English-Hindi and English-Spanish. We fine-tune multilingual models on artificially generated code-switched data.
  arXiv  Detail & Related papers  (2020-04-26T13:28:34Z)
• XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation [100.09099800591822]
  XGLUE is a new benchmark dataset that can be used to train large-scale cross-lingual pre-trained models. XGLUE provides 11 diversified tasks that cover both natural language understanding and generation scenarios.
  arXiv  Detail & Related papers  (2020-04-03T07:03:12Z)
• XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization [128.37244072182506]
  Cross-lingual TRansfer Evaluation of Multilinguals XTREME is a benchmark for evaluating the cross-lingual generalization capabilities of multilingual representations across 40 languages and 9 tasks. We demonstrate that while models tested on English reach human performance on many tasks, there is still a sizable gap in the performance of cross-lingually transferred models.
  arXiv  Detail & Related papers  (2020-03-24T19:09:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.