Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs

• URL: http://arxiv.org/abs/2608.18131v1
• Date: Sun, 26 Jul 2026 08:23:25 GMT
• Title: Safety Alignment Illusion: The Cross-Lingual Safety Gap in LLMs
• Abstract summary: Current safety alignment training for large language models are heavily English-centric.<n>This represents a critical failure mode for spoken language technologies deployed across India's linguistically diverse population.<n>For spoken language technologies deployed across India's linguistically diverse population, this represents a critical failure mode.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current safety alignment training for Large Language Models (LLMs) are heavily English-centric. When such safety filters fail for non-English languages, the consequences are immediate and user-facing: voice assistants and spoken dialogue systems may produce stereotype-reinforcing outputs, bypassing the standard English-focused safety alignments and propagating harmful bias to non-English speaking communities. For spoken language technologies deployed across India's linguistically diverse population, this represents a critical failure mode. To address this cross-lingual gap, we introduce INCLUDE (Indian Cultural Lens for Understanding and Detecting Embedded Biases), a multilingual evaluation benchmark designed to quantify Indian-centric socio-cultural biases. INCLUDE consists of 2,604 prompts spanning six prompt languages: English, Hindi, Bengali, Marathi, Tamil, and Hinglish (Hindi-English code-mix). We evaluate ten open- and closed-source LLMs against this benchmark, analyzing 14,988 bias scores. Our statistical results reveal two key findings. First, Bengali yielded the highest average bias score in open-source models. Second, English demonstrated a notable reversal, producing the lowest bias in open-source models but the highest bias in closed-source models.

Related papers

• Building Multilingual Bridges: Data Mixing as the Pillar of Generalization for In-Language Reasoning [75.46846056993418]
  Reasoning language models primarily reason in English regardless of the language they are prompted in.<n>This is inaccessible for non-English-speaking users, risks losing the intent of the original question, and forgoes knowledge more readily expressed in the target language.<n>In this work, we advance L2 reasoning, the ability of a model to reason consistently in the language of the user's prompt, thus building an in-language bridge between the prompt and the answer.
  arXiv  Detail & Related papers  (2026-09-09T16:56:25Z)
• IndicSafeEval: Safety Robustness of Large Language Models under Multilingual Persuasive Jailbreak Attacks [33.08360320836576]
  We introduce IndicSafeEval, a persuasion-based jailbreak evaluation framework for Indian languages.<n>Our benchmark combines ten safety critical content categories with six human-like persuasive strategies across four different Indian languages.<n>Our analysis shows that the model does not behave equally safely across all languages and prompt styles.
  arXiv  Detail & Related papers  (2026-09-03T12:54:44Z)
• SurakshaEval: An Indic Safety Benchmark for Multilingual LLMs [56.34354368047577]
  Existing safety evaluation datasets for large language models (LLMs) predominantly focus on English and Western contexts.<n>We introduce SurakshaEval, a novel safety benchmark composed of human-written prompts spanning real-world scenarios.<n>We benchmark a broad range of state-of-the-art LLMs on SurakshaEval, establish baseline safety performance, and identify recurring failure modes.
  arXiv  Detail & Related papers  (2026-08-08T02:03:33Z)
• Cross-Language Bias Examination in Large Language Models [37.21579885190632]
  This study introduces an innovative multilingual bias evaluation framework for assessing bias in Large Language Models.<n>By translating the prompts and word list into five target languages, we compare different types of bias across languages.<n>For example, Arabic and Spanish consistently show higher levels of stereotype bias, while Chinese and English exhibit lower levels of bias.
  arXiv  Detail & Related papers  (2025-12-17T23:22:03Z)
• Response-Based Knowledge Distillation for Multilingual Jailbreak Prevention Unwittingly Compromises Safety [3.8433556466595937]
  Large language models (LLMs) are increasingly deployed worldwide, yet their safety alignment remains predominantly English-centric.<n>We introduce a novel application of knowledge distillation (KD) in the context of multilingual jailbreak prevention.<n>We distill the refusal behaviors of a proprietary teacher model into three open-source student models: Meta-Llama-3-8B-Instruct, Gemma-2-2B-IT, and Qwen3-8B.
  arXiv  Detail & Related papers  (2025-12-08T06:48:17Z)
• Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs [32.12545369011503]
  We introduce DebateBias-8K, a new multilingual, debate-style benchmark to reveal how narrative bias appears in realistic generative settings.<n>Our dataset includes 8,400 structured debate prompts spanning four sensitive domains: women's rights, socioeconomic development, terrorism, and religion, across seven languages.<n>Results show that all models reproduce entrenched stereotypes despite safety alignment.
  arXiv  Detail & Related papers  (2025-11-03T03:25:40Z)
• BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context [36.56689822791777]
  Existing benchmarks, such as the Bias Benchmark for Question Answering (BBQ), primarily focus on Western contexts.<n>We introduce BharatBBQ, a culturally adapted benchmark designed to assess biases in Hindi, English, Marathi, Bengali, Tamil, Telugu, Odia, and Assamese.<n>Our dataset contains 49,108 examples in one language that are expanded using translation and verification to 392,864 examples in eight different languages.
  arXiv  Detail & Related papers  (2025-08-09T20:24:24Z)
• Crosslingual Reasoning through Test-Time Scaling [51.55526326294275]
  We find that scaling up inference compute for English-centric reasoning language models (RLMs) improves multilingual mathematical reasoning across many languages.<n>While English-centric RLM's CoTs are naturally predominantly English, they consistently follow a quote-and-think pattern to reason about quoted non-English inputs.<n>We observe poor out-of-domain reasoning generalization, in particular from STEM to cultural commonsense knowledge, even for English.
  arXiv  Detail & Related papers  (2025-05-08T16:50:06Z)
• Assessing Agentic Large Language Models in Multilingual National Bias [31.67058518564021]
  Cross-language disparities in reasoning-based recommendations remain largely unexplored.<n>This study is the first to address this gap.<n>We investigate multilingual bias in state-of-the-art LLMs by analyzing their responses to decision-making tasks across multiple languages.
  arXiv  Detail & Related papers  (2025-02-25T08:07:42Z)
• Dictionary Insertion Prompting for Multilingual Reasoning on Multilingual Large Language Models [52.00446751692225]
  We present a novel and simple yet effective method called textbfDictionary textbfInsertion textbfPrompting (textbfDIP) When providing a non-English prompt, DIP looks up a word dictionary and inserts words' English counterparts into the prompt for LLMs. It then enables better translation into English and better English model thinking steps which leads to obviously better results.
  arXiv  Detail & Related papers  (2024-11-02T05:10:50Z)
• Assessing Dialect Fairness and Robustness of Large Language Models in Reasoning Tasks [68.33068005789116]
  We introduce ReDial, a benchmark containing 1.2K+ parallel query pairs in Standardized English and AAVE.<n>We evaluate widely used models, including GPT, Claude, Llama, Mistral, and the Phi model families.<n>Our work establishes a systematic and objective framework for analyzing LLM bias in dialectal queries.
  arXiv  Detail & Related papers  (2024-10-14T18:44:23Z)
• Evaluating Implicit Bias in Large Language Models by Attacking From a Psychometric Perspective [66.34066553400108]
  We conduct a rigorous evaluation of large language models' implicit bias towards certain demographics.<n>Inspired by psychometric principles, we propose three attack approaches, i.e., Disguise, Deception, and Teaching.<n>Our methods can elicit LLMs' inner bias more effectively than competitive baselines.
  arXiv  Detail & Related papers  (2024-06-20T06:42:08Z)
• COLD: A Benchmark for Chinese Offensive Language Detection [54.60909500459201]
  We use COLDataset, a Chinese offensive language dataset with 37k annotated sentences. We also propose textscCOLDetector to study output offensiveness of popular Chinese language models. Our resources and analyses are intended to help detoxify the Chinese online communities and evaluate the safety performance of generative language models.
  arXiv  Detail & Related papers  (2022-01-16T11:47:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.