SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures

• URL: http://arxiv.org/abs/2605.02601v1
• Date: Mon, 04 May 2026 13:49:44 GMT
• Title: SemEval-2026 Task 7: Everyday Knowledge Across Diverse Languages and Cultures
• Abstract summary: The task data consist of an extended version of our manually constructed BLEnD benchmark (Myung et al. 2024), covering more than 30 language-culture pairs.<n>As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification.<n>We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches.
• Score: 93.36078244548077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present our shared task on evaluating the adaptability of LLMs and NLP systems across multiple languages and cultures. The task data consist of an extended version of our manually constructed BLEnD benchmark (Myung et al. 2024), covering more than 30 language-culture pairs, predominantly representing low-resource languages spoken across multiple continents. As the task is designed strictly for evaluation, participants were not permitted to use the data for training, fine-tuning, few-shot learning, or any other form of model modification. Our task includes two tracks: (a) Short-Answer Questions (SAQ) and (b) Multiple-Choice Questions (MCQ). Participants were required to predict labels and were allowed to submit any NLP system and adopt diverse modelling strategies, provided that the benchmark was used solely for evaluation. The task attracted more than 140 registered participants, and we received final submissions from 62 teams, along with 19 system description papers. We report the results and present an analysis of the best-performing systems and the most commonly adopted approaches. Furthermore, we discuss shared insights into open questions and challenges related to evaluation, misalignment, and methodological perspectives on model behaviour in low-resource languages and for under-represented cultures.

Related papers

• WorldBench: Culturally Grounded Benchmark for Multilingual Agents [69.76002914143531]
  Existing benchmarks rarely test state preservation, performance across languages, and application to realistic, grounded scenarios.<n>We present WorldBench: a comprehensive, multilingual benchmark, where agents can act in a sandbox via structured actions.<n>WorldBench comprises 1,600 tasks across seven languages and eight cultures, filtered and refined through feedback from human annotators.
  arXiv  Detail & Related papers  (2026-09-01T10:53:07Z)
• ImageEval 2026: Culturally Grounded Arabic Multimodal Evaluation [22.077150957957006]
  We present an overview of the ImageEval 2026 shared task on culturally grounded Arabic multimodal evaluation.<n>It includes two tasks: (i) AynVQA, covering spoken visual question answering and image-grounded hallucination detection in English and Modern Standard Arabic (MSA), and (ii) CRAI-Bench, evaluating the cultural accuracy of text-to-image generation.<n>A total of 14 teams participated in the test phase, with 12 teams submitting system description papers.
  arXiv  Detail & Related papers  (2026-08-31T09:02:28Z)
• EuraGovExam: A Multilingual Multimodal Benchmark from Real-World Civil Service Exams [0.8399688944263844]
  EuraGovExam is a benchmark sourced from real-world civil service examinations across five representative Eurasian regions.<n>The dataset contains over 8,000 high-resolution scanned multiple-choice questions covering 17 diverse academic and administrative domains.<n>It also supports practical applications in e-governance, public-sector document analysis, and equitable exam preparation.
  arXiv  Detail & Related papers  (2026-03-28T10:29:51Z)
• Challenging the Abilities of Large Language Models in Italian: a Community Initiative [63.94242079171895]
  "Challenging the Abilities of LAnguage Models in ITAlian" (CALAMITA) is a large-scale collaborative benchmarking initiative for Italian.<n>It federates more than 80 contributors from academia, industry, and the public sector to design, document, and evaluate a diverse collection of tasks.<n>We report results for four open-weight LLMs, highlighting systematic strengths and weaknesses across abilities.
  arXiv  Detail & Related papers  (2025-12-04T12:50:29Z)
• SLRTP2025 Sign Language Production Challenge: Methodology, Results, and Future Work [87.9341538630949]
  The first Sign Language Production Challenge was held as part of the third SLRTP Workshop at CVPR 2025.<n>The competition's aims are to evaluate architectures that translate from spoken language sentences to a sequence of skeleton poses.<n>This paper presents the challenge design and the winning methodologies.
  arXiv  Detail & Related papers  (2025-08-09T11:57:33Z)
• CEA-LIST at CheckThat! 2025: Evaluating LLMs as Detectors of Bias and Opinion in Text [3.9845507207125967]
  This paper presents a competitive approach to multilingual subjectivity detection using large language models (LLMs) with few-shot prompting.<n>We show that LLMs, when paired with carefully designed prompts, can match or outperform fine-tuned smaller language models (SLMs)<n>Our system achieved top rankings across multiple languages in the CheckThat! 2025 subjectivity detection task.
  arXiv  Detail & Related papers  (2025-07-10T08:35:05Z)
• SandboxAQ's submission to MRL 2024 Shared Task on Multi-lingual Multi-task Information Retrieval [1.2629889435114405]
  This paper explores the problems of Question Answering (QA) and Named Entity Recognition (NER) in five diverse languages. We tested five Large Language Models with various prompting methods, including zero-shot, chain-of-thought reasoning, and translation techniques. Our results show that while some models consistently outperform others, their effectiveness varies significantly across tasks and languages.
  arXiv  Detail & Related papers  (2024-10-28T20:15:45Z)
• From Multiple-Choice to Extractive QA: A Case Study for English and Arabic [51.13706104333848]
  We explore the feasibility of repurposing an existing multilingual dataset for a new NLP task.<n>We present annotation guidelines and a parallel EQA dataset for English and Modern Standard Arabic.<n>We aim to help others adapt our approach for the remaining 120 BELEBELE language variants, many of which are deemed under-resourced.
  arXiv  Detail & Related papers  (2024-04-26T11:46:05Z)
• The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics [36.52897053496835]
  generative large language models (LLMs) have shown remarkable capabilities to solve tasks with minimal or no task-related examples. We introduce the Eval4NLP 2023 shared task that asks participants to explore prompting and score extraction for machine translation (MT) and summarization evaluation. We present an overview of participants' approaches and evaluate them on a new reference-free test set spanning three language pairs for MT and a summarization dataset.
  arXiv  Detail & Related papers  (2023-10-30T17:55:08Z)
• Findings of the 2023 ML-SUPERB Challenge: Pre-Training and Evaluation over More Languages and Beyond [87.4049283495551]
  The 2023 Multilingual Speech Universal Performance Benchmark (ML-SUPERB) Challenge expands upon the acclaimed SUPERB framework.<n>The challenge garnered 12 model submissions and 54 language corpora, resulting in a comprehensive benchmark encompassing 154 languages.<n>The findings indicate that merely scaling models is not the definitive solution for multilingual speech tasks.
  arXiv  Detail & Related papers  (2023-10-09T08:30:01Z)
• Team QUST at SemEval-2023 Task 3: A Comprehensive Study of Monolingual and Multilingual Approaches for Detecting Online News Genre, Framing and Persuasion Techniques [0.030458514384586396]
  This paper describes the participation of team QUST in the SemEval2023 task 3. The monolingual models are first evaluated with the under-sampling of the majority classes. The pre-trained multilingual model is fine-tuned with a combination of the class weights and the sample weights.
  arXiv  Detail & Related papers  (2023-04-09T08:14:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.