What Matters When Building Universal Multilingual Named Entity Recognition Models?

• URL: http://arxiv.org/abs/2601.06347v1
• Date: Fri, 09 Jan 2026 23:02:37 GMT
• Title: What Matters When Building Universal Multilingual Named Entity Recognition Models?
• Authors: Jonas Golde, Patrick Haller, Alan Akbik,
• Abstract summary: We introduce Otter, a universal multilingual NER model supporting over 100 languages.<n>Otter achieves consistent improvements over strong multilingual NER baselines, outperforming GLiNER-x-base by 5.3pp in F1.<n>We release model checkpoints, training and evaluation code to facilitate and future research.
• Score: 12.125413756152833
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent progress in universal multilingual named entity recognition (NER) has been driven by advances in multilingual transformer models and task-specific architectures, loss functions, and training datasets. Despite substantial prior work, we find that many critical design decisions for such models are made without systematic justification, with architectural components, training objectives, and data sources evaluated only in combination rather than in isolation. We argue that these decisions impede progress in the field by making it difficult to identify which choices improve model performance. In this work, we conduct extensive experiments around architectures, transformer backbones, training objectives, and data composition across a wide range of languages. Based on these insights, we introduce Otter, a universal multilingual NER model supporting over 100 languages. Otter achieves consistent improvements over strong multilingual NER baselines, outperforming GLiNER-x-base by 5.3pp in F1 and achieves competitive performance compared to large generative models such as Qwen3-32B, while being substantially more efficient. We release model checkpoints, training and evaluation code to facilitate reproducibility and future research.

Related papers

• AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages [30.309928265469427]
  Large language models (LLMs) are increasingly multilingual, yet open models continue to underperform relative to proprietary systems.<n>We present textttAfriqueLLM, a suite of open LLMs adapted to 20 African languages through CPT on 26B tokens.
  arXiv  Detail & Related papers  (2026-01-10T02:39:31Z)
• TowerVision: Understanding and Improving Multilinguality in Vision-Language Models [56.775118098058506]
  TowerVision is a family of open multilingual vision-language models for both image-text and video-text tasks.<n>By incorporating visual and cultural context during fine-tuning, our models surpass existing approaches.<n>To support further research, we publicly release all models, data, and training recipes.
  arXiv  Detail & Related papers  (2025-10-22T17:02:48Z)
• RetrieveAll: A Multilingual Named Entity Recognition Framework with Large Language Models [7.867158538366131]
  Existing multilingual NER methods face severe language interference during the multi-language adaptation process.<n>We propose RetrieveAll, a universal multilingual NER framework based on dynamic LoRA.<n>We introduce a cross-granularity knowledge augmented method that fully exploits the intrinsic potential of the data.
  arXiv  Detail & Related papers  (2025-05-25T12:52:18Z)
• The Unreasonable Effectiveness of Model Merging for Cross-Lingual Transfer in LLMs [45.08958917457921]
  Large language models (LLMs) still struggle across tasks outside of high-resource languages.<n>In this work, we investigate cross-lingual transfer to lower-resource languages where task-specific post-training data is scarce.
  arXiv  Detail & Related papers  (2025-05-23T20:28:31Z)
• LOLA -- An Open-Source Massively Multilingual Large Language Model [1.5704590739448838]
  LOLA is a massively multilingual large language model trained on more than 160 languages.<n>Our architectural and implementation choices address the challenge of harnessing linguistic diversity.<n>We show how the learned expert-routing mechanism exploits implicit phylogenetic patterns to potentially alleviate the curse of multilinguality.
  arXiv  Detail & Related papers  (2024-09-17T15:23:08Z)
• IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities [4.269326314400742]
  We introduce the Inner-Adaptor Architecture for multimodal large language models (MLLMs)<n>The architecture incorporates multiple multimodal adaptors at varying depths within the large language model to facilitate direct interaction with the inherently text-oriented transformer layers.<n>Unlike previous approaches of freezing language models that require large-scale aligned data, our proposed architecture is able to achieve superior performance on small-scale datasets.
  arXiv  Detail & Related papers  (2024-08-23T08:10:13Z)
• What matters when building vision-language models? [52.8539131958858]
  We develop Idefics2, an efficient foundational vision-language model with 8 billion parameters. Idefics2 achieves state-of-the-art performance within its size category across various multimodal benchmarks. We release the model (base, instructed, and chat) along with the datasets created for its training.
  arXiv  Detail & Related papers  (2024-05-03T17:00:00Z)
• Assessing and Enhancing the Robustness of Large Language Models with Task Structure Variations for Logical Reasoning [25.496627355906966]
  We develop three new logical reasoning datasets named "ReClor-plus", "LogiQA-plus" and "LogiQAv2-plus"<n>Experiments show that these simple augmentations greatly hinder the models' performance.<n>Applying logic-driven data augmentation for fine-tuning and prompting can enhance generalisation in both discriminative and generative models.
  arXiv  Detail & Related papers  (2023-10-13T22:29:15Z)
• Improving Massively Multilingual ASR With Auxiliary CTC Objectives [40.10307386370194]
  We introduce our work on improving performance on FLEURS, a 102-language open ASR benchmark. We investigate techniques inspired from recent Connectionist Temporal Classification ( CTC) studies to help the model handle the large number of languages. Our state-of-the-art systems using self-supervised models with the Conformer architecture improve over the results of prior work on FLEURS by a relative 28.4% CER.
  arXiv  Detail & Related papers  (2023-02-24T18:59:51Z)
• Language Models are General-Purpose Interfaces [109.45478241369655]
  We propose to use language models as a general-purpose interface to various foundation models. A collection of pretrained encoders perceive diverse modalities (such as vision, and language) We propose a semi-causal language modeling objective to jointly pretrain the interface and the modular encoders.
  arXiv  Detail & Related papers  (2022-06-13T17:34:22Z)
• Distributionally Robust Multilingual Machine Translation [94.51866646879337]
  We propose a new learning objective for Multilingual neural machine translation (MNMT) based on distributionally robust optimization. We show how to practically optimize this objective for large translation corpora using an iterated best response scheme. Our method consistently outperforms strong baseline methods in terms of average and per-language performance under both many-to-one and one-to-many translation settings.
  arXiv  Detail & Related papers  (2021-09-09T03:48:35Z)
• GroupBERT: Enhanced Transformer Architecture with Efficient Grouped Structures [57.46093180685175]
  We demonstrate a set of modifications to the structure of a Transformer layer, producing a more efficient architecture. We add a convolutional module to complement the self-attention module, decoupling the learning of local and global interactions. We apply the resulting architecture to language representation learning and demonstrate its superior performance compared to BERT models of different scales.
  arXiv  Detail & Related papers  (2021-06-10T15:41:53Z)
• Learning Universal Representations from Word to Sentence [89.82415322763475]
  This work introduces and explores the universal representation learning, i.e., embeddings of different levels of linguistic unit in a uniform vector space. We present our approach of constructing analogy datasets in terms of words, phrases and sentences. We empirically verify that well pre-trained Transformer models incorporated with appropriate training settings may effectively yield universal representation.
  arXiv  Detail & Related papers  (2020-09-10T03:53:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.