Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset

• URL: http://arxiv.org/abs/2601.00411v1
• Date: Thu, 01 Jan 2026 17:53:38 GMT
• Title: Do LLMs Judge Distantly Supervised Named Entity Labels Well? Constructing the JudgeWEL Dataset
• Authors: Alistair Plum, Laura Bernardy, Tharindu Ranasinghe,
• Abstract summary: We present judgeWEL, a dataset for named entity recognition (NER) in Luxembourgish, automatically labelled and verified using large language models (LLM)<n>By exploiting internal links within Wikipedia articles, we infer entity types based on their corresponding Wikidata entries.<n>Because such links are not uniformly reliable, we mitigate noise by employing and comparing several LLMs to identify and retain only high-quality labelled sentences.
• Score: 8.437906092903582
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present judgeWEL, a dataset for named entity recognition (NER) in Luxembourgish, automatically labelled and subsequently verified using large language models (LLM) in a novel pipeline. Building datasets for under-represented languages remains one of the major bottlenecks in natural language processing, where the scarcity of resources and linguistic particularities make large-scale annotation costly and potentially inconsistent. To address these challenges, we propose and evaluate a novel approach that leverages Wikipedia and Wikidata as structured sources of weak supervision. By exploiting internal links within Wikipedia articles, we infer entity types based on their corresponding Wikidata entries, thereby generating initial annotations with minimal human intervention. Because such links are not uniformly reliable, we mitigate noise by employing and comparing several LLMs to identify and retain only high-quality labelled sentences. The resulting corpus is approximately five times larger than the currently available Luxembourgish NER dataset and offers broader and more balanced coverage across entity categories, providing a substantial new resource for multilingual and low-resource NER research.

Related papers

• BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR [0.06363400715351396]
  This work presents a Bangla IR dataset constructed using a BETA-labeling framework.<n>We examine whether IR datasets from other low-resource languages can be effectively reused through one-hop machine translation.
  arXiv  Detail & Related papers  (2026-02-16T06:04:04Z)
• Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains [6.357124887141297]
  Large Language Models (LLMs) have redefined Machine Translation (MT)<n>LLMs often exhibit uneven performance across language families and specialized domains.<n>We introduce Translation Tangles, a unified framework and dataset for evaluating the translation quality and fairness of open-source LLMs.
  arXiv  Detail & Related papers  (2025-10-09T07:28:30Z)
• LexMatcher: Dictionary-centric Data Collection for LLM-based Machine Translation [67.24113079928668]
  We present LexMatcher, a method for data curation driven by the coverage of senses found in bilingual dictionaries. Our approach outperforms the established baselines on the WMT2022 test sets.
  arXiv  Detail & Related papers  (2024-06-03T15:30:36Z)
• GlotLID: Language Identification for Low-Resource Languages [51.38634652914054]
  GlotLID-M is an LID model that satisfies the desiderata of wide coverage, reliability and efficiency. It identifies 1665 languages, a large increase in coverage compared to prior work.
  arXiv  Detail & Related papers  (2023-10-24T23:45:57Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• CLSE: Corpus of Linguistically Significant Entities [58.29901964387952]
  We release a Corpus of Linguistically Significant Entities (CLSE) annotated by experts. CLSE covers 74 different semantic types to support various applications from airline ticketing to video games. We create a linguistically representative NLG evaluation benchmark in three languages: French, Marathi, and Russian.
  arXiv  Detail & Related papers  (2022-11-04T12:56:12Z)
• CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation [113.99145386490639]
  Cross-lingual NER can transfer knowledge between languages via aligned cross-lingual representations or machine translation results. We propose a Cross-lingual Entity Projection framework (CROP) to enable zero-shot cross-lingual NER. We adopt a multilingual labeled sequence translation model to project the tagged sequence back to the target language and label the target raw sentence.
  arXiv  Detail & Related papers  (2022-10-13T13:32:36Z)
• DaN+: Danish Nested Named Entities and Lexical Normalization [18.755176247223616]
  This paper introduces DaN+, a new multi-domain corpus and annotation guidelines for Danish nested named entities (NEs) and lexical normalization. We empirically assess three strategies to model the two-layer Named Entity Recognition (NER) task. Our results show that 1) the most robust strategy is multi-task learning which is rivaled by multi-label decoding, 2) BERT-based NER models are sensitive to domain shifts, and 3) in-language BERT and lexical normalization are the most beneficial on the least canonical data.
  arXiv  Detail & Related papers  (2021-05-24T14:35:21Z)
• Named Entity Recognition and Linking Augmented with Large-Scale Structured Data [3.211619859724085]
  We describe our submissions to the 2nd and 3rd SlavNER Shared Tasks held at BSNLP 2019 and BSNLP 2021. The tasks focused on the analysis of Named Entities in multilingual Web documents in Slavic languages with rich inflection. Our solution takes advantage of large collections of both unstructured and structured documents.
  arXiv  Detail & Related papers  (2021-04-27T20:10:18Z)
• Multilingual Autoregressive Entity Linking [49.35994386221958]
  mGENRE is a sequence-to-sequence system for the Multilingual Entity Linking problem. For a mention in a given language, mGENRE predicts the name of the target entity left-to-right, token-by-token. We show the efficacy of our approach through extensive evaluation including experiments on three popular MEL benchmarks.
  arXiv  Detail & Related papers  (2021-03-23T13:25:55Z)
• AMALGUM -- A Free, Balanced, Multilayer English Web Corpus [14.073494095236027]
  We present a genre-balanced English web corpus totaling 4M tokens. By tapping open online data sources the corpus is meant to offer a more sizable alternative to smaller manually created annotated data sets.
  arXiv  Detail & Related papers  (2020-06-18T17:05:45Z)
• Soft Gazetteers for Low-Resource Named Entity Recognition [78.00856159473393]
  We propose a method of "soft gazetteers" that incorporates ubiquitously available information from English knowledge bases into neural named entity recognition models. Our experiments on four low-resource languages show an average improvement of 4 points in F1 score.
  arXiv  Detail & Related papers  (2020-05-04T21:58:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.