XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity

• URL: http://arxiv.org/abs/2605.05662v1
• Date: Thu, 07 May 2026 04:35:03 GMT
• Title: XL-SafetyBench: A Country-Grounded Cross-Cultural Benchmark for LLM Safety and Cultural Sensitivity
• Abstract summary: XL-SafetyBench is a suite of 5,500 test cases across 10 country-language pairs.<n>It consists of a Jailbreak Benchmark of country-grounded adversarial prompts and a Cultural Benchmark where local sensitivities are embedded within innocuous requests.<n>Each item is constructed via a multi-stage pipeline that combines LLM-assisted principled discovery, automated validation gates, and dual independent native-speaker annotators per country.
• Score: 3.342964361434013
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current LLM safety benchmarks are predominantly English-centric and often rely on translation, failing to capture country-specific harms. Moreover, they rarely evaluate a model's ability to detect culturally embedded sensitivities as distinct from universal harms. We introduce XL-SafetyBench. a suite of 5,500 test cases across 10 country-language pairs, comprising a Jailbreak Benchmark of country-grounded adversarial prompts and a Cultural Benchmark where local sensitivities are embedded within innocuous requests. Each item is constructed via a multi-stage pipeline that combines LLM-assisted discovery, automated validation gates, and dual independent native-speaker annotators per country. To distinguish principled refusal from comprehension failure, we evaluate Attack Success Rate (ASR) alongside two complementary metrics we introduce: Neutral-Safe Rate (NSR) and Cultural Sensitivity Rate (CSR). Evaluating 10 frontier and 27 local LLMs reveals two key findings. First, jailbreak robustness and cultural awareness do not show a coupled relationship among frontier models, so a composite safety score obscures per-axis variation. Second, local models exhibit a near-linear ASR-NSR trade-off (r = -0.81), indicating that their apparent safety reflects generation failure rather than genuine alignment. XL-SafetyBench enables more nuanced, cross-cultural safety evaluation in the multilingual era.

Related papers

• SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs [56.34354368047577]
  Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts.<n>We introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios.<n>We benchmark a broad range of state-of-the-art LLMs on SurakshaEval, establish baseline safety performance, and identify recurring failure modes.
  arXiv  Detail & Related papers  (2026-08-08T02:03:33Z)
• CHILLGuard: Towards Fine-Grained Chinese LLM Safety Guardrail with Scalable Data Construction and Model-aware Preference Alignment [55.74660714572696]
  Malicious content generated from large language models (LLMs) could pose severe safety risks and ethical concerns.<n>Existing LLM safety guardrails excel in English or multilingual settings, but lack adaptation to Chinese-specific regulatory policies, cultural context and linguistic nuances.<n>We introduce a 5-macro, 31-micro category fine-grained risk taxonomy for Chinese scenarios, and build CHILLGuard: a dedicated Chinese LLM content safety guardrail.
  arXiv  Detail & Related papers  (2026-06-13T16:57:51Z)
• Schützen: Evaluating LLM Safety in Bulgarian and German Contexts [53.865251738592605]
  This paper introduces Schtzen: a German--Bulgarian safety dataset designed to assess model answerability under risk.<n> Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior.
  arXiv  Detail & Related papers  (2026-06-09T18:01:19Z)
• Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese [1.0221857326642831]
  ChiSafe-PAS is a human-annotated benchmark of 1,897 adversarial Chinese prompts.<n>It covers four high-stakes domains: self-harm and violence, drug and illicit trade, fraud, and satire.<n>We describe the dataset design, annotation process, and obfuscation taxonomy.
  arXiv  Detail & Related papers  (2026-05-28T09:28:51Z)
• KSAFE-MM: A Multimodal Safety Benchmark via Localized Contextualization for Korean Cultural Risks [11.227509564873621]
  This paper introduces KSAFE-MM, a benchmark for Korean multimodal safety evaluation.<n>KSAFE-MM consists of two parts, KSAFE-MM-G and KSAFE-MM-C.<n>We evaluate 12 state-of-the-art MLLMs on KSAFE-MM and reveal that models exhibit greater vulnerability to culturally grounded attacks than to generic ones.
  arXiv  Detail & Related papers  (2026-05-27T06:08:54Z)
• ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety [2.929429337200341]
  Safety evaluations for large language models increasingly target high-stakes National Security and Public Safety (NSPS) risks.<n> empirical evidence of how language and geopolitical context interact remains limited to a narrow set of language pairs.<n>We introduce emphROK-FORTRESS, a bilingual, culturally adversarial NSPS benchmark that uses the English--Korean language pair and U.S.--ROK geopolitical axis as a case study.
  arXiv  Detail & Related papers  (2026-05-13T22:07:22Z)
• IndicSafe: A Benchmark for Evaluating Multilingual LLM Safety in South Asia [0.6978180153516672]
  We present the first systematic evaluation of large language models (LLMs) safety across 12 Indic languages.<n>Cross-language agreement is just 12.8%, and textttSAFE rate variance exceeds 17% across languages.<n>We release textscIndicSafe, the first benchmark to enable culturally informed safety evaluation for Indic deployments.
  arXiv  Detail & Related papers  (2026-03-18T16:54:07Z)
• Can Large Language Models Make Everyone Happy? [12.59854280011403]
  Misalignment in Large Language Models (LLMs) refers to the failure to simultaneously satisfy safety, value, and cultural dimensions.<n>We introduce MisAlign-Profile, a unified benchmark for measuring misalignment trade-offs inspired by mechanistic profiling.
  arXiv  Detail & Related papers  (2026-02-11T17:57:23Z)
• SalamahBench: Toward Standardized Safety Evaluation for Arabic Language Models [6.0860786631767185]
  This paper introduces SalamaBench, a unified benchmark for evaluating the safety of Arabic Language Models (ALMs)<n>Using this benchmark, we evaluate five state-of-the-art ALMs, including Fanar 1 and 2, ALLaM 2, Falcon H1R, and Jais 2, under multiple safeguard configurations.
  arXiv  Detail & Related papers  (2026-02-03T12:13:42Z)
• SEA-SafeguardBench: Evaluating AI Safety in SEA Languages and Cultures [36.95168918567729]
  Existing multilingual safety benchmarks often rely on machine-translated English data.<n>We introduce SEA-SafeguardBench, the first human-verified safety benchmark for SEA.<n>It covers eight languages, 21,640 samples, across three subsets: general, in-the-wild, and content generation.
  arXiv  Detail & Related papers  (2025-12-05T07:57:57Z)
• OutSafe-Bench: A Benchmark for Multimodal Offensive Content Detection in Large Language Models [54.80460603255789]
  We introduce OutSafe-Bench, the first most comprehensive content safety evaluation test suite designed for the multimodal era.<n>OutSafe-Bench includes a large-scale dataset that spans four modalities, featuring over 18,000 bilingual (Chinese and English) text prompts, 4,500 images, 450 audio clips and 450 videos, all systematically annotated across nine critical content risk categories.<n>In addition to the dataset, we introduce a Multidimensional Cross Risk Score (MCRS), a novel metric designed to model and assess overlapping and correlated content risks across different categories.
  arXiv  Detail & Related papers  (2025-11-13T13:18:27Z)
• Which Cultural Lens Do Models Adopt? On Cultural Positioning Bias and Agentic Mitigation in LLMs [53.07843733899881]
  Large language models (LLMs) have unlocked a wide range of downstream generative applications.<n>We find that they also risk perpetuating subtle fairness issues tied to culture, positioning their generations from the perspectives of the mainstream US culture.<n>We propose 2 inference-time mitigation methods to resolve these biases.
  arXiv  Detail & Related papers  (2025-09-25T12:28:25Z)
• XGUARD: A Graded Benchmark for Evaluating Safety Failures of Large Language Models on Extremist Content [3.4303348430261202]
  We present XGUARD, a benchmark and evaluation framework designed to assess the severity of extremist content generated by Large Language Models (LLMs)<n>XGUARD includes 3,840 red teaming prompts sourced from real world data such as social media and news, covering a broad range of ideologically charged scenarios.<n>Our framework categorizes model responses into five danger levels (0 to 4), enabling a more nuanced analysis of both the frequency and severity of failures.
  arXiv  Detail & Related papers  (2025-06-01T11:48:54Z)
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming [64.86326523181553]
  ALERT is a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models.
  arXiv  Detail & Related papers  (2024-04-06T15:01:47Z)
• Fake Alignment: Are LLMs Really Aligned Well? [91.26543768665778]
  This study investigates the substantial discrepancy in performance between multiple-choice questions and open-ended questions. Inspired by research on jailbreak attack patterns, we argue this is caused by mismatched generalization.
  arXiv  Detail & Related papers  (2023-11-10T08:01:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.