When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs

• URL: http://arxiv.org/abs/2607.21445v1
• Date: Thu, 23 Jul 2026 15:52:00 GMT
• Title: When Trivia Is Not Trivial: Everyday Knowledge Failures in Multilingual LLMs
• Abstract summary: We introduce TriviaRoomQA, a benchmark designed to evaluate everyday, culturally grounded, and long-tail knowledge.<n>We evaluate 30 open-weight LLMs from European, Asian, and North American providers, covering models from 7 to 70B parameters.<n>We find that models are strong on knowledge-intensive topics such as history, geography, and mathematics, but substantially weaker on everyday popular-culture topics such as celebrities, music, movies, and news.
• Score: 4.942653708116484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quiz rooms, trivia nights, and quiz shows challenge human knowledge across a wide range of topics, from canonical facts to everyday culture. In this paper, we examine whether large language models (LLMs) can perform competitively in such settings, using quiz-style questions to test them on both common and niche topics. We introduce TriviaRoomQA, a multilingual benchmark designed to evaluate everyday, culturally grounded, and long-tail knowledge across 288 topics. The benchmark contains 3,300 parallel multiple-choice questions in six European languages and additional 5,340 French-only questions for a more fine-grained case study. We evaluate 30 open-weight LLMs from European, Asian, and North American providers, covering models from 7 to 70B parameters. We find that models are strong on knowledge-intensive topics such as history, geography, and mathematics, but substantially weaker on everyday popular-culture topics such as celebrities, music, movies, and news. Moreover, model performance varies across languages even for the same underlying questions, suggesting that access to factual knowledge is not always language-independent. In sum, our dataset and experiments demonstrate an important knowledge gap which is not captured by existing academic-based saturated benchmarks.

Related papers

• RespondeoQA: a Benchmark for Bilingual Latin-English Question Answering [2.5578258168516816]
  We introduce a benchmark dataset for question answering and translation in bilingual Latin and English settings.<n>The questions are drawn from Latin pedagogical sources, including exams, quizbowl-style trivia, and textbooks ranging from the 1800s to the present.<n>The dataset covers a diverse range of question types: knowledge- and skill-based, multihop reasoning, constrained translation, and mixed language pairs.
  arXiv  Detail & Related papers  (2026-04-22T16:24:46Z)
• Leveraging Wikidata for Geographically Informed Sociocultural Bias Dataset Creation: Application to Latin America [4.587479593949085]
  Large Language Models (LLMs) are trained on Global North data and show prejudicial behavior towards other cultures.<n>We propose to leverage the content of Wikipedia, the structure of the Wikidata knowledge graph, and expert knowledge from social science in order to create a dataset of question/answer pairs.<n>We create the LatamQA database of over 26k questions and associated answers extracted from 26k Wikipedia articles, and transformed into multiple-choice questions (MCQ) in Spanish and Portuguese, in turn translated to English.
  arXiv  Detail & Related papers  (2026-02-16T14:23:17Z)
• Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge [68.6805229085352]
  Most multilingual question-answering benchmarks do not factor in regional diversity in the information they capture.<n>XNationQA encompasses a total of 49,280 questions on the geography, culture, and history of nine countries, presented in seven languages.<n>We benchmark eight standard multilingual LLMs on XNationQA and evaluate them using two novel transference metrics.
  arXiv  Detail & Related papers  (2025-11-01T18:41:34Z)
• SinhalaMMLU: A Comprehensive Benchmark for Evaluating Multitask Language Understanding in Sinhala [39.525952729268994]
  We introduce SinhalaMMLU, the first multiple-choice question answering benchmark designed specifically for Sinhala.<n>The dataset includes over 7,000 questions spanning secondary to collegiate education levels, aligned with the Sri Lankan national curriculum.<n>We evaluate 26 LLMs on SinhalaMMLU and observe that, while Claude 3.5 sonnet and GPT-4o achieve the highest average accuracies at 67% and 62% respectively, overall model performance remains limited.
  arXiv  Detail & Related papers  (2025-09-03T09:22:39Z)
• ParamBench: A Graduate-Level Benchmark for Evaluating LLM Understanding on Indic Subjects [4.2155105586549535]
  We present ParamBench, consisting of more than 17K questions in the Hindi language, comprising questionnaires from 21 diverse subjects.<n>These questions are primarily derived from a nationwide graduate-level entrance examination covering topics such as history, music, instruments, yoga,ush, literature, philosophy, law, etc.<n>We evaluate the performance of more than 16 open source LLMs on this benchmark, observing that Gemma3-27B attains the highest overall accuracy of 56.4%.
  arXiv  Detail & Related papers  (2025-08-22T07:59:37Z)
• WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines [74.25764182510295]
  Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English.<n>We introduce World Cuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding.<n>This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points.
  arXiv  Detail & Related papers  (2024-10-16T16:11:49Z)
• CaLMQA: Exploring culturally specific long-form question answering across 23 languages [58.18984409715615]
  CaLMQA is a dataset of 51.7K culturally specific questions across 23 different languages.<n>We evaluate factuality, relevance and surface-level quality of LLM-generated long-form answers.
  arXiv  Detail & Related papers  (2024-06-25T17:45:26Z)
• BertaQA: How Much Do Language Models Know About Local Culture? [33.27901483916244]
  We introduce BertaQA, a trivia dataset parallel in English and Basque. The dataset consists of a local subset with questions pertinent to the Basque culture, and a global subset with questions of broader interest. We show that continued pre-training in Basque significantly improves the models' performance on Basque culture, even when queried in English.
  arXiv  Detail & Related papers  (2024-06-11T14:30:34Z)
• CVQA: Culturally-diverse Multilingual Visual Question Answering Benchmark [68.21939124278065]
  Culturally-diverse multilingual Visual Question Answering benchmark designed to cover a rich set of languages and cultures. CVQA includes culturally-driven images and questions from across 30 countries on four continents, covering 31 languages with 13 scripts, providing a total of 10k questions. We benchmark several Multimodal Large Language Models (MLLMs) on CVQA, and show that the dataset is challenging for the current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-06-10T01:59:00Z)
• MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models [65.10456412127405]
  MLaKE is a benchmark for the adaptability of knowledge editing methods across five languages.<n>MLaKE aggregates fact chains from Wikipedia across languages and generates questions in both free-form and multiple-choice.<n>We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE.
  arXiv  Detail & Related papers  (2024-04-07T15:23:28Z)
• M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models [76.88692952308084]
  M3Exam is a benchmark for evaluating large language models (LLMs) in a multilingual, multimodal, and multilevel context. M3Exam contains 12,317 questions in 9 diverse languages with three educational levels. We assess the performance of top-performing LLMs on M3Exam and find that current models, including GPT-4, still struggle with multilingual text.
  arXiv  Detail & Related papers  (2023-06-08T13:21:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.