Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese

• URL: http://arxiv.org/abs/2606.07300v1
• Date: Fri, 05 Jun 2026 14:17:09 GMT
• Title: Phun-Bench: Evaluating LLMs on Phonological Understanding in Chinese
• Abstract summary: Existing benchmarks on phonological abilities are either solvable through rote memorization or intertwined with other abilities, making them inadequate to measure phonological understanding.<n>Here, we present Phun-Bench, a purpose-built Chinese benchmark with diverse tasks and settings across three dimensions (Homophony, Rhyme, and Phonetic Similarity)<n>Our results show that while LLMs excel at recalling correct pronunciations, they generally struggle to leverage phonological knowledge in the flexible and intuitive way that human speakers do.
• Score: 21.39607432360906
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language is a vehicle for thought, intricately tied to sounds, symbols, and meaning. However, most large language model (LLM) research focuses on meaning (semantics) and symbols (spelling) while largely overlooking sounds. Existing benchmarks on LLMs' phonological abilities are either solvable through rote memorization or intertwined with other abilities, making them inadequate to measure LLMs' genuine ability in phonological understanding. Here, we present Phun-Bench, a purpose-built Chinese benchmark with diverse tasks and settings across three dimensions (Homophony, Rhyme, and Phonetic Similarity), designed to systematically evaluate LLMs' phonological understanding. Our results show that while LLMs excel at recalling correct pronunciations, they generally struggle to leverage phonological knowledge in the flexible and intuitive way that human speakers do. Moreover, through detailed analyses, we propose a hypothesis regarding the underlying mechanism of LLMs' phonological understanding and "perception", highlighting an underexplored frontier for future research.

Related papers

• Large Language Models as Modal Models in Linguistics [10.50113943900077]
  We argue that large language models (LLMs) possess genuine epistemic value as minimal models.<n>We then examine the conditions under which LLMs could qualify as how-actually explanations (HAEs) of human language.
  arXiv  Detail & Related papers  (2026-06-09T06:37:24Z)
• Do Language Models Associate Sound with Meaning? A Multimodal Study of Sound Symbolism [20.62188582405012]
  We investigate how Multimodal Large Language Models (MLLMs) interpret auditory information in human languages.<n>We present LEX-ICON, an extensive mimetic word dataset consisting of 8,052 words from four natural languages.<n>Key findings demonstrate (1) MLLMs' phonetic intuitions that align with existing linguistic research across multiple semantic dimensions and (2) phonosemantic attention patterns that highlight models' focus on iconic phonemes.
  arXiv  Detail & Related papers  (2025-11-13T07:46:09Z)
• SpeechIQ: Speech Intelligence Quotient Across Cognitive Levels in Voice Understanding Large Language Models [76.07833875692722]
  Speech-based Intelligence Quotient (SIQ) is a new form of human cognition-inspired evaluation pipeline for voice understanding large language models.<n>Our framework represents a first-of-its-kind intelligence examination that bridges cognitive principles with voice-oriented benchmarks.
  arXiv  Detail & Related papers  (2025-07-25T15:12:06Z)
• On the Semantics of Large Language Models [0.0]
  Large Language Models (LLMs) demonstrated the potential to replicate human language abilities through technology.<n>It remains controversial to what extent these systems truly understand language.<n>We examine this issue by narrowing the question down to the semantics of LLMs at the word and sentence level.
  arXiv  Detail & Related papers  (2025-07-07T20:02:57Z)
• PhoniTale: Phonologically Grounded Mnemonic Generation for Typologically Distant Language Pairs [51.745816131869674]
  Large language models (LLMs) have been used to generate keyword mnemonics by leveraging similar keywords from a learner's first language (L1) to aid in acquiring L2 vocabulary.<n>We present PhoniTale, a novel cross-lingual mnemonic generation system that performs IPA-based phonological adaptation and syllable-aware alignment to retrieve L1 keyword sequence.<n>Our findings show that PhoniTale consistently outperforms previous automated approaches and achieves quality comparable to human-written mnemonics.
  arXiv  Detail & Related papers  (2025-07-07T19:50:12Z)
• XToM: Exploring the Multilingual Theory of Mind for Large Language Models [57.9821865189077]
  Existing evaluations of Theory of Mind in LLMs are largely limited to English.<n>We present XToM, a rigorously validated multilingual benchmark that evaluates ToM across five languages.<n>Our findings expose limitations in LLMs' ability to replicate human-like mentalizing across linguistic contexts.
  arXiv  Detail & Related papers  (2025-06-03T05:23:25Z)
• When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners [111.50503126693444]
  We show that language-specific ablation consistently boosts multilingual reasoning performance.<n>Compared to post-training, our training-free ablation achieves comparable or superior results with minimal computational overhead.
  arXiv  Detail & Related papers  (2025-05-21T08:35:05Z)
• Large Language Models as Neurolinguistic Subjects: Discrepancy between Performance and Competence [49.60849499134362]
  This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning)<n>We introduce a neurolinguistic approach, utilizing a novel method that combines minimal pair and diagnostic probing to analyze activation patterns across model layers.<n>We found: (1) Psycholinguistic and neurolinguistic methods reveal that language performance and competence are distinct; (2) Direct probability measurement may not accurately assess linguistic competence; and (3) Instruction tuning won't change much competence but improve performance.
  arXiv  Detail & Related papers  (2024-11-12T04:16:44Z)
• PhonologyBench: Evaluating Phonological Skills of Large Language Models [57.80997670335227]
  Phonology, the study of speech's structure and pronunciation rules, is a critical yet often overlooked component in Large Language Model (LLM) research. We present PhonologyBench, a novel benchmark consisting of three diagnostic tasks designed to explicitly test the phonological skills of LLMs. We observe a significant gap of 17% and 45% on Rhyme Word Generation and Syllable counting, respectively, when compared to humans.
  arXiv  Detail & Related papers  (2024-04-03T04:53:14Z)
• Can phones, syllables, and words emerge as side-products of cross-situational audiovisual learning? -- A computational investigation [2.28438857884398]
  We study the so-called latent language hypothesis (LLH) LLH connects linguistic representation learning to general predictive processing within and across sensory modalities. We explore LLH further in extensive learning simulations with different neural network models for audiovisual cross-situational learning.
  arXiv  Detail & Related papers  (2021-09-29T05:49:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.