YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese

• URL: http://arxiv.org/abs/2607.00664v1
• Date: Wed, 01 Jul 2026 09:13:19 GMT
• Title: YOMI-Bench: A Benchmark for Evaluating Kanji Reading and Phonological Understanding of LLMs for Japanese
• Abstract summary: In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone.<n>The proposed YOMI-Bench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese.
• Score: 24.032881992865697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose YOMI-Bench, a benchmark for evaluating kanji reading and phonological understanding of large language models (LLMs) for Japanese. In Japanese, a single kanji character often has multiple possible readings, making it difficult to infer the correct reading from surface-level text alone. Due to these linguistic characteristics, it is empirically known that LLMs exhibit low performance in kanji reading for Japanese. The proposed YOMI-Bench consists of four tasks specifically designed to evaluate kanji reading performance in Japanese. In our evaluation using YOMI-Bench, we assessed one multilingual open LLM, four Japanese-specific open LLMs, and five commercial LLMs. As a result, we found that even Japanese-specific models show low performance, and that commercial models also perform poorly on generation tasks that require consideration of kanji readings.

Related papers

• MultiNRC: A Challenging and Native Multilingual Reasoning Evaluation Benchmark for LLMs [56.87573414161703]
  We introduce the Multilingual Native Reasoning Challenge (MultiNRC), a benchmark to assess Large Language Models (LLMs)<n>MultiNRC covers four core reasoning categories: language-specific linguistic reasoning, wordplay & riddles, cultural/tradition reasoning, and math reasoning with cultural relevance.<n>For cultural/tradition reasoning and math reasoning with cultural relevance, we also provide English equivalent translations of the multilingual questions by manual translation from native speakers fluent in English.
  arXiv  Detail & Related papers  (2025-07-23T12:56:31Z)
• Automatically Suggesting Diverse Example Sentences for L2 Japanese Learners Using Pre-Trained Language Models [29.400532173806834]
  This study examines the use of Pre-trained Language Models (PLMs) to produce example sentences targeting L2 Japanese learners.<n>We evaluate the quality of sentences by considering multiple aspects such as difficulty, diversity, and naturalness, with a panel of raters consisting of learners of Japanese, native speakers -- and GPT-4.
  arXiv  Detail & Related papers  (2025-06-04T05:13:05Z)
• EXECUTE: A Multilingual Benchmark for LLM Token Understanding [54.70665106141121]
  Tests across multiple languages reveal that challenges in other languages are not always on the character level as in English.<n>We also examine sub-character tasks in Chinese, Japanese, and Korean to assess LLMs' understanding of character components.
  arXiv  Detail & Related papers  (2025-05-23T11:56:48Z)
• PolyMath: Evaluating Mathematical Reasoning in Multilingual Contexts [85.78821098963607]
  PolyMath is a multilingual mathematical reasoning benchmark covering 18 languages and 4 easy-to-hard difficulty levels.<n>Our benchmark ensures difficulty comprehensiveness, language diversity, and high-quality translation.
  arXiv  Detail & Related papers  (2025-04-25T15:39:04Z)
• Why We Build Local Large Language Models: An Observational Analysis from 35 Japanese and Multilingual LLMs [26.737806062804506]
  We evaluated 35 Japanese, English, and multilingual LLMs on 19 evaluation benchmarks for Japanese and English.<n>We found that training on English text can improve the scores of academic subjects in Japanese.<n>It is unnecessary to specifically train on Japanese text to enhance abilities for solving Japanese code generation, arithmetic reasoning, commonsense, and reading comprehension tasks.
  arXiv  Detail & Related papers  (2024-12-19T02:39:26Z)
• JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation [63.83457341009046]
  JMMMU (Japanese MMMU) is the first large-scale Japanese benchmark designed to evaluate LMMs on expert-level tasks based on the Japanese cultural context.<n>Using the CA subset, we observe performance drop in many LMMs when evaluated in Japanese, which is purely attributable to language variation.<n>By combining both subsets, we identify that some LMMs perform well on the CA subset but not on the CS subset, exposing a shallow understanding of the Japanese language that lacks depth in cultural understanding.
  arXiv  Detail & Related papers  (2024-10-22T17:59:56Z)
• MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models [65.10456412127405]
  MLaKE is a benchmark for the adaptability of knowledge editing methods across five languages.<n>MLaKE aggregates fact chains from Wikipedia across languages and generates questions in both free-form and multiple-choice.<n>We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE.
  arXiv  Detail & Related papers  (2024-04-07T15:23:28Z)
• CMMLU: Measuring massive multitask language understanding in Chinese [133.70911295934746]
  This paper introduces a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.
  arXiv  Detail & Related papers  (2023-06-15T15:49:51Z)
• Exploration of Language Dependency for Japanese Self-Supervised Speech Representation Models [18.22157315310462]
  Self-supervised learning (SSL) has been dramatically successful not only in monolingual but also in cross-lingual settings. In this paper, we investigate how effective a cross-lingual model is in comparison with a monolingual model. We examine how much unlabeled data collected in Japanese is needed to achieve performance comparable to a cross-lingual model pre-trained with tens of thousands of hours of English and/or multilingual data.
  arXiv  Detail & Related papers  (2023-05-09T06:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.