NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap

• URL: http://arxiv.org/abs/2608.04397v1
• Date: Wed, 05 Aug 2026 02:58:53 GMT
• Title: NOLLI: A Difficulty-Calibrated Puzzle Benchmark for Diagnosing the English-Korean Performance Gap
• Abstract summary: NOLLI is a benchmark designed to diagnose where Korean performance gaps arise.<n>It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution.<n>It combines matched direct translations, script adaptations over Hangul jamo (sub-syllabic letters), and Korean-only tasks grounded in Korean culture or orthography.
• Score: 6.166565297478229
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce NOLLI, a procedurally generated English-Korean puzzle benchmark designed to diagnose where Korean performance gaps arise. It comprises 15 puzzle types (25 tasks; 7,500 items), with every instance seed-regenerable, verified to have a unique solution, and scored deterministically. Rather than equating harder with bigger, we calibrate difficulty behaviorally, tuning each generator until a fixed reference model lands in target accuracy bands. Its three-level design combines matched direct translations, script adaptations over Hangul jamo (sub-syllabic letters), and Korean-only tasks grounded in Korean culture or orthography. We evaluate 15 frontier, open-weight, and Korean-developed models; among the 12 above a 3% overall-accuracy floor, matched English-Korean accuracy is statistically equivalent within a +/- 10 pp margin (TOST), suggesting little cost from presentation language alone. Writing-system-intensive tasks show sharper gaps: Korean Cipher falls behind English by up to 68.7 pp, whereas Cryptarithmetic over the same jamo shows no systematic penalty, and Jamo Composition accuracy predicts Korean Cipher accuracy. These contrasts are diagnostic rather than causal, consistent with difficulty in multi-step sub-syllabic execution. Korean-only tasks separate rule-application deficits, which vary in sign, from a Kinship deficit positive in all 12. Finally, a salient size measure fails to grow from Easy to Hard in 7 of 15 types, making structural size an unreliable proxy for empirical difficulty.

Related papers

• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Extending Item Response Theory for Efficient and Meaningful Multilingual Evaluation [19.965981631741894]
  Multilingual benchmarks are central to evaluating large language models (LLMs) across languages.<n>They suffer from three issues: exhaustive evaluation scales linearly with the number of languages, automatic translation introduces errors that are easily missed at scale, and some items conflate general and culture-specific knowledge.<n>We address all three with a unified statistical framework, Multilingual-IRT, which extends Item Response Theory with per-language difficulty deviations.
  arXiv  Detail & Related papers  (2026-06-14T07:16:03Z)
• K-BrowseComp: A Web Browsing Agent Benchmark Grounded in Korean Contexts [43.3460021951349]
  We introduce K-BrowseComp, a web-browsing agent benchmark grounded in Korean contexts, consisting of 400 problems.<n>The 300-problem K-BrowseComp-Verified subset is manually constructed and validated by native Korean speakers.<n>On this subset, frontier LLMs, including GPT-5.5, DeepSeek-V4-Pro, and GLM-5.1, reach only 30.00--45.67%, a substantial drop from BrowseComp.
  arXiv  Detail & Related papers  (2026-06-01T15:50:03Z)
• The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models [51.61416200800499]
  GaoYao is a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas.<n>First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers.<n>Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages.<n>Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs.
  arXiv  Detail & Related papers  (2026-04-22T06:19:46Z)
• KMMMU: Evaluation of Massive Multi-discipline Multimodal Understanding in Korean Language and Context [8.518732134730355]
  We introduce KMMMU, a native Korean benchmark for evaluating multimodal understanding in Korean cultural and institutional settings.<n>KMMMU contains 3,466 questions from exams written in Korean, covering nine disciplines and nine visual modality categories, along with a 300-item Korean-specific subset and a hard subset of 627 questions.<n>Experiments show that the strongest open-source model reaches only 42.05% accuracy on the full set, while the best proprietary model achieves 52.42% on the hard subset.
  arXiv  Detail & Related papers  (2026-03-18T01:58:14Z)
• KoBALT: Korean Benchmark For Advanced Linguistic Tasks [0.6971903955510721]
  KoBALT (Korean Benchmark for Advanced Linguistic Tasks) is a linguistically-motivated benchmark comprising 700 multiple-choice questions.<n>It is designed to advance the evaluation of large language models (LLMs) in Korean.<n>It introduces a suite of expert-curated, linguistically motivated questions with minimal n-gram overlap with standard Korean corpora.
  arXiv  Detail & Related papers  (2025-05-22T02:03:07Z)
• Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap [0.0]
  Large language models (LLMs) demonstrate exceptional performance on complex reasoning tasks.<n>Despite strong reasoning capabilities in high-resource languages, a significant performance gap persists in other languages.<n>We propose UST (Understand, Solve, and Translate), a method that strategically uses English as an anchor for reasoning and solution generation.
  arXiv  Detail & Related papers  (2025-01-05T05:57:22Z)
• Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping [60.458273797431836]
  Decoding by contrasting layers (DoLa) is designed to improve the generation quality of large language models. We find that this approach does not work well on non-English tasks. Inspired by previous interpretability work on language transition during the model's forward pass, we propose an improved contrastive decoding algorithm.
  arXiv  Detail & Related papers  (2024-07-15T15:14:01Z)
• On the Off-Target Problem of Zero-Shot Multilingual Neural Machine Translation [104.85258654917297]
  We find that failing in encoding discriminative target language signal will lead to off-target and a closer lexical distance. We propose Language Aware Vocabulary Sharing (LAVS) to construct the multilingual vocabulary. We conduct experiments on a multilingual machine translation benchmark in 11 languages.
  arXiv  Detail & Related papers  (2023-05-18T12:43:31Z)
• Making Large Language Models Better Reasoners with Step-Aware Verifier [49.16750018427259]
  DIVERSE (Diverse Verifier on Reasoning Step) is a novel approach that further enhances the reasoning capability of language models. We evaluate DIVERSE on the latest language model code-davinci and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks.
  arXiv  Detail & Related papers  (2022-06-06T03:38:36Z)
• Few-shot Learning with Multilingual Language Models [66.49496434282564]
  We train multilingual autoregressive language models on a balanced corpus covering a diverse set of languages. Our largest model sets new state of the art in few-shot learning in more than 20 representative languages. We present a detailed analysis of where the model succeeds and fails, showing in particular that it enables cross-lingual in-context learning.
  arXiv  Detail & Related papers  (2021-12-20T16:52:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.