Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety

• URL: http://arxiv.org/abs/2607.10112v1
• Date: Sat, 11 Jul 2026 04:13:42 GMT
• Title: Minionese: Comprehensive Benchmark and Mechanistic Study of Multilingual LLM Safety
• Abstract summary: Safety alignment in large language models remains brittle across languages.<n>We introduce textscMinionese, a multilingual jailbreak benchmark spanning 18 languages.<n>We show that each attack type produces a distinct vulnerability profile.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safety alignment in large language models remains brittle across languages: prompts reliably refused in English can elicit harmful compliance in non-English and low-resource settings. We introduce \textsc{Minionese}, a multilingual jailbreak benchmark spanning 18 languages, 4 resource tiers, and 4 perturbation types (standard translation, code-switching, transliteration, and translationese), paired with a geometric mechanistic analysis of refusal failure across language tiers. We show that each attack type produces a distinct vulnerability profile: transliteration vulnerability is mediated by script identity, code-switching maintains effectiveness through the lowest-resource tier, and a sharp safety regime transition between Tiers 2 and 3 is consistent across all models. Mechanistically, low-resource jailbreaks succeed by routing harmful content through a geometrically misaligned subspace that projects insufficiently onto the refusal directions, leaving the refusal mechanism intact but untriggered. These findings show that English-only safety evaluations are insufficient; they require accounting for script family, perturbation type, and per-language alignment coverage. The benchmark and analysis code is at https://github.com/Brentkong/Minionese-Comprehensive-Benchmark-and-Mechanistic-Study-of-Multilingual -LLM-Safety.git.

Related papers

• The Illusion of Cross-Lingual Safety in Low-Resource Languages [7.471112976694495]
  Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings.<n>We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts.<n>Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs.
  arXiv  Detail & Related papers  (2026-08-11T17:05:26Z)
• LLM Safety Alignment in Low-Resource Languages: A Systematic Literature Review [9.895109697164283]
  Large Language Models (LLMs) have achieved substantial progress in safety alignment.<n>Their safety guarantees remain significantly weaker in low-resource and multilingual settings.
  arXiv  Detail & Related papers  (2026-07-20T21:11:00Z)
• Why Do Safety Guardrails Degrade Across Languages? [21.521293656854183]
  Large language models exhibit safety degradation in non-English languages.<n>We introduce a latent variable model, a Multi-Group Item Response Theory (IRT) framework, that decouples safety-driving factors.<n>We evaluate the safety robustness of 61 model configurations across 5 closed-model families and 10 languages of varying resource.
  arXiv  Detail & Related papers  (2026-05-16T22:08:54Z)
• Cross-Lingual Jailbreak Detection via Semantic Codebooks [0.0]
  Safety mechanisms for large language models (LLMs) remain predominantly English-centric, creating systematic vulnerabilities in multilingual deployment.<n>We investigate whether such attacks can be mitigated through language-agnostic semantic similarity without retraining or language-specific adaptation.
  arXiv  Detail & Related papers  (2026-04-28T14:43:40Z)
• LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety [88.98698230989186]
  Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities in low-resource languages.<n>We attribute this gap to a mismatch between language-agnostic semantic understanding ability and language-dominant safety alignment biased toward high-resource languages.<n>We propose Language-Agnostic Semantic Alignment (LASA), which anchors safety alignment directly in semantic bottlenecks.
  arXiv  Detail & Related papers  (2026-04-13T15:59:50Z)
• Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages [57.059267233093465]
  Large Language Models (LLMs) have transformed natural language processing, but their safety mechanisms remain under-explored in low-resource, multilingual settings.<n>We introduce textsfSGToxicGuard, a novel dataset and evaluation framework for benchmarking LLM safety in Singapore's diverse linguistic context.<n>We conduct extensive experiments with state-of-the-art multilingual LLMs, and the results uncover critical gaps in their safety guardrails.
  arXiv  Detail & Related papers  (2025-09-18T08:14:34Z)
• RabakBench: Scaling Human Annotations to Construct Localized Multilingual Safety Benchmarks for Low-Resource Languages [3.7678366606419345]
  RabakBench is a new multilingual safety benchmark localized to Singapore's unique linguistic context.<n>The benchmark dataset, including the human-verified translations, and evaluation code are publicly available.
  arXiv  Detail & Related papers  (2025-07-08T13:37:25Z)
• Multilingual Collaborative Defense for Large Language Models [39.28665703568305]
  One notable vulnerability is the ability to bypass safeguards by translating harmful queries into rare or underrepresented languages.<n>Despite the growing concern, there has been limited research addressing the safeguarding of LLMs in multilingual scenarios.<n>We propose Multilingual Collaborative Defense (MCD), a novel learning method that optimize a continuous, soft safety prompt automatically.
  arXiv  Detail & Related papers  (2025-05-17T04:47:16Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• LLMs Lost in Translation: M-ALERT uncovers Cross-Linguistic Safety Inconsistencies [63.10843814055688]
  M-ALERT is a benchmark that evaluates the safety of Large Language Models in five languages.<n>M-ALERT includes 15k high-quality prompts per language, totaling 75k, with category-wise annotations.<n>Our experiments on 39 state-of-the-art LLMs highlight the importance of language-specific safety analysis.
  arXiv  Detail & Related papers  (2024-12-19T16:46:54Z)
• Benchmarking LLM Guardrails in Handling Multilingual Toxicity [57.296161186129545]
  We introduce a comprehensive multilingual test suite, spanning seven datasets and over ten languages, to benchmark the performance of state-of-the-art guardrails. We investigate the resilience of guardrails against recent jailbreaking techniques, and assess the impact of in-context safety policies and language resource availability on guardrails' performance. Our findings show that existing guardrails are still ineffective at handling multilingual toxicity and lack robustness against jailbreaking prompts.
  arXiv  Detail & Related papers  (2024-10-29T15:51:24Z)
• Multilingual Jailbreak Challenges in Large Language Models [96.74878032417054]
  In this study, we reveal the presence of multilingual jailbreak challenges within large language models (LLMs) We consider two potential risky scenarios: unintentional and intentional. We propose a novel textscSelf-Defense framework that automatically generates multilingual training data for safety fine-tuning.
  arXiv  Detail & Related papers  (2023-10-10T09:44:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.