Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

• URL: http://arxiv.org/abs/2605.29667v1
• Date: Thu, 28 May 2026 09:28:51 GMT
• Title: Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese
• Abstract summary: ChiSafe-PAS is a human-annotated benchmark of 1,897 adversarial Chinese prompts.<n>It covers four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire.<n>We describe the dataset design, annotation process, and obfuscation taxonomy.
• Score: 1.0221857326642831
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When Large Language Models (LLMs) are deployed in Chinese-language settings, a troubling pattern emerges: safety systems that work well in English break down. These systems struggle to cross linguistic and cultural bound-aries, leaving models exposed to adversarial prompts that exploit Chinese-specific evasion techniques, including Pinyin romanization, character decomposition, internet slang, and hedging tone. To address this gap, we introduce ChiSafe-PAS (Chinese Safety Pilot Annotation Set), a human-annotated benchmark of 1,897 adversarial Chinese prompts spanning four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire. Of these, 1,544 entries carry complete gold-standard annotations: a 3-class response label (REFUSE, SAFE-REDIRECT, RESPOND), a nine-category obfuscation taxonomy, a risk-level rating, and annotator rationale. We describe the dataset design, annotation process, and obfuscation taxonomy in detail. Our primary goal is practical: to give the research community a high-quality, culturally grounded resource for benchmarking LLM safety alignment. In doing so, we engage three broader tensions in the field: the blurring boundary between training and evaluation data, the need for domain coverage grounded in real-world risk, and the limits of scale as a substitute for cultural expertise.

Related papers

• SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs [56.34354368047577]
  Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts.<n>We introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios.<n>We benchmark a broad range of state-of-the-art LLMs on SurakshaEval, establish baseline safety performance, and identify recurring failure modes.
  arXiv  Detail & Related papers  (2026-08-08T02:03:33Z)
• CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment [55.74660714572696]
  Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.<n>Existing LLM safety guardrails excel in English or multilingual settings, but lack adaptation to Chinese-specific regulatory policies, cultural context and linguistic nuances.<n>We introduce a 5-macro, 31-micro category fine-grained risk taxonomy for Chinese scenarios, and build CHILLGuard: a dedicated Chinese LLM content safety guardrail.
  arXiv  Detail & Related papers  (2026-06-13T16:57:51Z)
• Schützen: Evaluating LLM Safety in Bulgarian and German Contexts [53.865251738592605]
  This paper introduces Schtzen: a German--Bulgarian safety dataset designed to assess model answerability under risk.<n> Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior.
  arXiv  Detail & Related papers  (2026-06-09T18:01:19Z)
• IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia [0.6978180153516672]
  We present the first systematic evaluation of large language models (LLMs) safety across 12 Indic languages.<n>Cross-language agreement is just 12.8%, and textttSAFE rate variance exceeds 17% across languages.<n>We release textscIndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments.
  arXiv  Detail & Related papers  (2026-03-18T16:54:07Z)
• Improving Methodologies for LLM Evaluations Across Global Languages [19.63570354411416]
  The exercise shows how safety behaviours can vary across languages.<n>It also generated insights for improving multilingual safety evaluations.<n>This work represents an initial step toward a shared framework for multilingual safety testing of advanced AI systems.
  arXiv  Detail & Related papers  (2026-01-22T07:18:08Z)
• CSSBench: Evaluating the Safety of Lightweight LLMs against Chinese-Specific Adversarial Patterns [43.09570813322625]
  We introduce the Chinese-Specific Safety Benchmark (CSSBench) to evaluate the safety of large language models (LLMs) in Chinese.<n>Our benchmark covers six domains that are common in real Chinese scenarios, including illegal activities and compliance, privacy leakage, health and medical misinformation, fraud and hate, and public and political safety.<n>Our results show that the Chinese-specific adversarial pattern is a critical challenge for lightweight LLMs.
  arXiv  Detail & Related papers  (2026-01-02T06:21:41Z)
• Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages [57.059267233093465]
  Large Language Models (LLMs) have transformed natural language processing, but their safety mechanisms remain under-explored in low-resource, multilingual settings.<n>We introduce textsfSGToxicGuard, a novel dataset and evaluation framework for benchmarking LLM safety in Singapore's diverse linguistic context.<n>We conduct extensive experiments with state-of-the-art multilingual LLMs, and the results uncover critical gaps in their safety guardrails.
  arXiv  Detail & Related papers  (2025-09-18T08:14:34Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• ChineseSafe: A Chinese Benchmark for Evaluating Safety in Large Language Models [13.911977148887873]
  We present a Chinese safety benchmark (ChineseSafe) to facilitate research on the content safety of large language models.<n>Our ChineseSafe contains 205,034 examples across 4 classes and 10 sub-classes of safety issues.<n>For Chinese contexts, we add several special types of illegal content: political sensitivity, pornography, and variant/homophonic words.
  arXiv  Detail & Related papers  (2024-10-24T07:25:29Z)
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming [64.86326523181553]
  ALERT is a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models.
  arXiv  Detail & Related papers  (2024-04-06T15:01:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.