Evaluating Chinese Ambiguity Understanding in Large Language Models

• URL: http://arxiv.org/abs/2605.15635v1
• Date: Fri, 15 May 2026 05:35:18 GMT
• Title: Evaluating Chinese Ambiguity Understanding in Large Language Models
• Abstract summary: CHA-Gen is the first PA Theory-grounded Chinese ambiguity dataset.<n>It comprises 5,712 sentences (2,414 ambiguous, 3,298 unambiguous) across 18 potential ambiguous structures.
• Score: 15.610863020079327
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Linguistic ambiguity is critical to the robustness of Large Language Models (LLMs), yet existing research focuses mostly on English, with limited attention devoted to Chinese. Existing Chinese ambiguity datasets (e.g., CHAmbi) suffer from poor scalability. Guided by Potential Ambiguity (PA) Theory, we design a semi-automatic pipeline to construct CHA-Gen. It is the first PA Theory-grounded Chinese ambiguity dataset, which comprises 5,712 sentences (2,414 ambiguous, 3,298 unambiguous) across 18 potential ambiguous structures. Evaluating LLMs (e.g. Gemma 3, Qwen 2.5/3 series) via direct querying and machine translation, we find that LLMs struggle with ambiguity detection (improved by CoT prompting). Analysis of Qwen3-32B's CoT rationales reveals three common failure modes: ambiguity blindness, misattribution, and premature resolution. Uncertainty quantification with semantic entropy metric shows higher uncertainty for ambiguous sentences. Moreover, instruction tuning induces overconfidence, whereas Base models better capture semantic diversity. We further observe that models exhibit a bias toward dominant interpretations. Our work provides a scalable approach for Chinese ambiguity corpus and insights into LLMs' ambiguity handling, laying a foundation for enhancing Chinese ambiguity research in LLMs.

Related papers

• A Multimodal Dataset for Visually Grounded Ambiguity in Machine Translation [53.06004926782424]
  We present VIDA (Visually-Dependent Ambiguity), a dataset of 2,500 carefully curated instances in which resolving an annotated source span requires visual evidence.<n>We also propose Disambiguation-Centric Metrics that use an LLM-as-a-judge classifier to verify whether annotated ambiguous expressions are resolved correctly at the span level.
  arXiv  Detail & Related papers  (2026-05-03T19:55:06Z)
• It Depends: Resolving Referential Ambiguity in Minimal Contexts with Commonsense Knowledge [3.340255811686752]
  We investigate whether Large Language Models can leverage commonsense to resolve referential ambiguity in multi-turn conversations.<n>We test DeepSeek v3, GPT-4o, Qwen3-32B, GPT-4o-mini, and Llama-3.1-8B via LLM-as-Judge and human annotations.
  arXiv  Detail & Related papers  (2025-09-19T15:49:26Z)
• mSCoRe: a $M$ultilingual and Scalable Benchmark for $S$kill-based $Co$mmonsense $Re$asoning [74.97363626515236]
  We propose a textbfMultilingual and Scalable Benchmark for textbfSkill-based textbfCommonsense textbfReasoning (textbfmSCoRe)<n>Our benchmark incorporates three key components that are designed to systematically evaluate LLM's reasoning capabilities.<n>Our results reveal the limitations of such reasoning-reinforced models when confronted with nuanced multilingual general and cultural commonsense.
  arXiv  Detail & Related papers  (2025-08-13T18:59:02Z)
• Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity [16.065963688326242]
  We study the trustworthiness of large language models (LLMs) when encountering ambiguous narrative text in Chinese.<n>We created a benchmark dataset by collecting and generating ambiguous sentences with context and their corresponding disambiguated pairs.<n>We discovered significant fragility in LLMs when handling ambiguity, revealing behavior that differs substantially from humans.
  arXiv  Detail & Related papers  (2025-07-30T21:50:19Z)
• Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons [43.708431369382176]
  We introduce a small challenge dataset for NLI with large lexical overlap. We show that GPT-4 and Llama 2 fail it with strong bias. From a Computational Linguistics perspective, we identify a group of constructions with three classes of adjectives which cannot be distinguished by surface features.
  arXiv  Detail & Related papers  (2024-03-26T14:51:12Z)
• Towards Effective Disambiguation for Machine Translation with Large Language Models [65.80775710657672]
  We study the capabilities of large language models to translate "ambiguous sentences" Experiments show that our methods can match or outperform state-of-the-art systems such as DeepL and NLLB in four out of five language directions.
  arXiv  Detail & Related papers  (2023-09-20T22:22:52Z)
• Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models [27.491408293411734]
  Large Language Models (LLMs) show promising results in language generation and instruction following but frequently "hallucinate" Our research introduces a simple redundancy: not all tokens in auto-regressive text equally represent the underlying meaning.
  arXiv  Detail & Related papers  (2023-07-03T22:17:16Z)
• Simple Linguistic Inferences of Large Language Models (LLMs): Blind Spots and Blinds [59.71218039095155]
  We evaluate language understanding capacities on simple inference tasks that most humans find trivial. We target (i) grammatically-specified entailments, (ii) premises with evidential adverbs of uncertainty, and (iii) monotonicity entailments. The models exhibit moderate to low performance on these evaluation sets.
  arXiv  Detail & Related papers  (2023-05-24T06:41:09Z)
• We're Afraid Language Models Aren't Modeling Ambiguity [136.8068419824318]
  Managing ambiguity is a key part of human language understanding. We characterize ambiguity in a sentence by its effect on entailment relations with another sentence. We show that a multilabel NLI model can flag political claims in the wild that are misleading due to ambiguity.
  arXiv  Detail & Related papers  (2023-04-27T17:57:58Z)
• Is Supervised Syntactic Parsing Beneficial for Language Understanding? An Empirical Investigation [71.70562795158625]
  Traditional NLP has long held (supervised) syntactic parsing necessary for successful higher-level semantic language understanding (LU) Recent advent of end-to-end neural models, self-supervised via language modeling (LM), and their success on a wide range of LU tasks, questions this belief. We empirically investigate the usefulness of supervised parsing for semantic LU in the context of LM-pretrained transformer networks.
  arXiv  Detail & Related papers  (2020-08-15T21:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.