The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation

• URL: http://arxiv.org/abs/2604.00019v1
• Date: Wed, 11 Mar 2026 01:02:55 GMT
• Title: The Chronicles of RiDiC: Generating Datasets with Controlled Popularity Distribution for Long-form Factuality Evaluation
• Abstract summary: The RiDiC dataset contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers.<n>Each entity is accompanied by its geographical location, English and Chinese names, and relevant English and Chinese Wikipedia content.<n>These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate.
• Score: 56.22548620937169
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a configurable pipeline for generating multilingual sets of entities with specified characteristics, such as domain, geographical location and popularity, using data from Wikipedia and Wikidata. These datasets are intended for evaluating the factuality of LLMs' long-form generation, thereby complementing evaluation based on short-form QA datasets. We present the RiDiC dataset as an example of this approach. RiDiC contains 3,000 entities from three domains -- rivers, natural disasters, and car models -- spanning different popularity tiers. Each entity is accompanied by its geographical location, English and Chinese names (if available) and relevant English and Chinese Wikipedia content, which is used to evaluate LLMs' responses. Generations about RiDiC entities were obtained from three LLMs in English and Chinese. These were then evaluated using a third-party factuality checker, which showed that entities from our dataset caused even frontier models to hallucinate. To facilitate the evaluation of LLMs' long-form factuality in multiple languages, the code, data, and generation/evaluation scripts have been released.

Related papers

• MORE: A Multilingual Document Parsing Benchmark and Evaluation [13.789730225958877]
  MORE is a large-scale benchmark designed for multilingual document parsing evaluation.<n>It covers 149 languages, making it the most linguistically diverse benchmark to date.<n>All samples are curated from real-world documents via a model-assisted, human-refined annotation pipeline.
  arXiv  Detail & Related papers  (2026-07-03T04:59:12Z)
• Towards Reliable Multilingual LLMs-as-a-Judge: An Empirical Study [15.5103844307963]
  Large language models (LLMs) are increasingly used for the automatic evaluation of generated text.<n>This work explores several strategies for developing multilingual LLMs-as-a-judge.
  arXiv  Detail & Related papers  (2026-05-27T16:33:58Z)
• HPLT 3.0: Very Large-Scale Multilingual Resources for LLM and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models [25.953042884928006]
  We present an initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages.<n>At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data.<n>We train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models.
  arXiv  Detail & Related papers  (2025-11-02T20:16:38Z)
• Ready to Translate, Not to Represent? Bias and Performance Gaps in Multilingual LLMs Across Language Families and Domains [6.357124887141297]
  Large Language Models (LLMs) have redefined Machine Translation (MT)<n>LLMs often exhibit uneven performance across language families and specialized domains.<n>We introduce Translation Tangles, a unified framework and dataset for evaluating the translation quality and fairness of open-source LLMs.
  arXiv  Detail & Related papers  (2025-10-09T07:28:30Z)
• Evaluating Large Language Model with Knowledge Oriented Language Specific Simple Question Answering [73.73820209993515]
  We introduce KoLasSimpleQA, the first benchmark evaluating the multilingual factual ability of Large Language Models (LLMs)<n>Inspired by existing research, we created the question set with features such as single knowledge point coverage, absolute objectivity, unique answers, and temporal stability.<n>Results show significant performance differences between the two domains.
  arXiv  Detail & Related papers  (2025-05-22T12:27:02Z)
• A Comprehensive Survey on Long Context Language Modeling [118.5540791080351]
  Long Context Language Models (LCLMs) process and analyze extensive inputs in an effective and efficient way.<n>Our survey is structured around three key aspects: how to obtain effective and efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate and analyze LCLMs comprehensively.
  arXiv  Detail & Related papers  (2025-03-20T17:06:28Z)
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages [86.90220551111096]
  Training datasets for large language models (LLMs) are often not fully disclosed. We present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages.
  arXiv  Detail & Related papers  (2023-09-17T23:49:10Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• Building Multilingual Corpora for a Complex Named Entity Recognition and Classification Hierarchy using Wikipedia and DBpedia [0.0]
  We present the UNER dataset, a multilingual and hierarchical parallel corpus annotated for named-entities. We describe in detail the developed procedure necessary to create this type of dataset in any language available on Wikipedia with DBpedia information.
  arXiv  Detail & Related papers  (2022-12-14T11:38:48Z)
• STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation [48.56586847883825]
  We introduce a dataset and evaluation platform built from STORIUM, an online collaborative storytelling community. Our dataset contains 6K lengthy stories with fine-grained natural language annotations interspersed throughout each narrative. We evaluate language models fine-tuned on our dataset by integrating them onto STORIUM, where real authors can query a model for suggested story continuations and then edit them.
  arXiv  Detail & Related papers  (2020-10-04T23:26:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.