LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models

• URL: http://arxiv.org/abs/2508.12733v2
• Date: Wed, 27 Aug 2025 12:21:20 GMT
• Title: LinguaSafe: A Comprehensive Multilingual Safety Benchmark for Large Language Models
• Authors: Zhiyuan Ning, Tianle Gu, Jiaxin Song, Shixin Hong, Lingyu Li, Huacan Liu, Jie Li, Yixu Wang, Meng Lingyu, Yan Teng, Yingchun Wang,
• Abstract summary: Our dataset comprises 45k entries in 12 languages, ranging from Hungarian to Malay.<n>Our benchmark provides a comprehensive suite of metrics for in-depth safety evaluation.
• Score: 22.273388934888278
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The widespread adoption and increasing prominence of large language models (LLMs) in global technologies necessitate a rigorous focus on ensuring their safety across a diverse range of linguistic and cultural contexts. The lack of a comprehensive evaluation and diverse data in existing multilingual safety evaluations for LLMs limits their effectiveness, hindering the development of robust multilingual safety alignment. To address this critical gap, we introduce LinguaSafe, a comprehensive multilingual safety benchmark crafted with meticulous attention to linguistic authenticity. The LinguaSafe dataset comprises 45k entries in 12 languages, ranging from Hungarian to Malay. Curated using a combination of translated, transcreated, and natively-sourced data, our dataset addresses the critical need for multilingual safety evaluations of LLMs, filling the void in the safety evaluation of LLMs across diverse under-represented languages from Hungarian to Malay. LinguaSafe presents a multidimensional and fine-grained evaluation framework, with direct and indirect safety assessments, including further evaluations for oversensitivity. The results of safety and helpfulness evaluations vary significantly across different domains and different languages, even in languages with similar resource levels. Our benchmark provides a comprehensive suite of metrics for in-depth safety evaluation, underscoring the critical importance of thoroughly assessing multilingual safety in LLMs to achieve more balanced safety alignment. Our dataset and code are released to the public to facilitate further research in the field of multilingual LLM safety.

Related papers

• The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It [21.6479207553511]
  We review nearly 300 publications from 2020--2024 across major NLP conferences and workshops at *ACL.<n>We observe that non-English languages are rarely studied as a standalone language and that English safety research exhibits poor language documentation practice.<n>Based on our survey and proposed directions, the field can develop more robust, inclusive AI safety practices for diverse global populations.
  arXiv  Detail & Related papers  (2025-05-30T01:32:44Z)
• MPO: Multilingual Safety Alignment via Reward Gap Optimization [88.76638442683391]
  Large language models (LLMs) have become increasingly central to AI applications worldwide.<n>Existing preference learning methods for safety alignment, such as RLHF and DPO, are primarily monolingual and struggle with noisy multilingual data.<n>We introduce Multilingual reward gaP Optimization (MPO), a novel approach that leverages the well-aligned safety capabilities of the dominant language (English) to improve safety alignment across multiple languages.
  arXiv  Detail & Related papers  (2025-05-22T16:24:51Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• PolyGuard: A Multilingual Safety Moderation Tool for 17 Languages [27.318299273902984]
  PolyGUARD is a new state-of-the-art multilingual safety model for safeguarding Large Language Models (LLMs) generations.<n>It is trained on the largest multilingual safety training corpus to date containing 1.91M samples across 17 languages.<n>PolyGUARDPROMPTS is a high quality multilingual benchmark with 29K samples for the evaluation of safety guardrails.
  arXiv  Detail & Related papers  (2025-04-06T06:09:21Z)
• LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies [63.10843814055688]
  M-ALERT is a benchmark that evaluates the safety of Large Language Models in five languages.<n>M-ALERT includes 15k high-quality prompts per language, totaling 75k, with category-wise annotations.<n>Our experiments on 39 state-of-the-art LLMs highlight the importance of language-specific safety analysis.
  arXiv  Detail & Related papers  (2024-12-19T16:46:54Z)
• Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixture [6.17896401271963]
  We introduce Multilingual Blending, a mixed-language query-response scheme designed to evaluate the safety alignment of various large language models. We investigate language patterns such as language availability, morphology, and language family that could impact the effectiveness of Multilingual Blending.
  arXiv  Detail & Related papers  (2024-07-10T03:26:15Z)
• ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming [64.86326523181553]
  ALERT is a large-scale benchmark to assess safety based on a novel fine-grained risk taxonomy. It aims to identify vulnerabilities, inform improvements, and enhance the overall safety of the language models.
  arXiv  Detail & Related papers  (2024-04-06T15:01:47Z)
• All Languages Matter: On the Multilingual Safety of Large Language Models [96.47607891042523]
  We build the first multilingual safety benchmark for large language models (LLMs) XSafety covers 14 kinds of commonly used safety issues across 10 languages that span several language families. We propose several simple and effective prompting methods to improve the multilingual safety of ChatGPT.
  arXiv  Detail & Related papers  (2023-10-02T05:23:34Z)
• SafetyBench: Evaluating the Safety of Large Language Models [54.878612385780805]
  SafetyBench is a comprehensive benchmark for evaluating the safety of Large Language Models (LLMs) It comprises 11,435 diverse multiple choice questions spanning across 7 distinct categories of safety concerns. Our tests over 25 popular Chinese and English LLMs in both zero-shot and few-shot settings reveal a substantial performance advantage for GPT-4 over its counterparts.
  arXiv  Detail & Related papers  (2023-09-13T15:56:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.