Language-Specific Gaps in AI Safety Training Datasets

• URL: http://arxiv.org/abs/2608.13695v1
• Date: Thu, 13 Aug 2026 18:40:22 GMT
• Title: Language-Specific Gaps in AI Safety Training Datasets
• Abstract summary: Self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied.<n>We show that self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model providers routinely cite multilingual safety benchmarks spanning a dozen or more languages as evidence that their models are safe for non-English-speaking users. We show that these collection-level coverage claims frequently do not survive inspection at the level of an individual language. Auditing 21 resources across 25 language slices, of which 20 count as datasets under our counting rules, spanning three languages chosen to represent low- (Hausa), mid- (Swahili), and high-resource (French) tiers, we find that gaps in provenance, annotation reliability, access, harm-taxonomy coverage, and data reuse recur in patterns that partially, but not fully, track resource level. Using a controlled within-pipeline comparison, we show a Hausa-language slice falling below its own paper's translation-quality acceptance threshold while the same pipeline's Swahili output clears the same bar comfortably; this is evidence that these gaps are measurable and addressable, not inherent. We further show that self-harm and sexual-content categories have no native-language coverage in either African-language tier we studied, a total rather than gradated gap that a purely resource-level account does not predict. We connect these findings to a documented, persistent asymmetry in multilingual jailbreak robustness (single-turn attacks largely mitigated, multi-turn attacks still effective), arguing that this asymmetry is structurally consistent with where our audit finds training and evaluation data thinnest. We contribute a reusable slice-level audit methodology, a cross-tier empirical comparison, and concrete recommendations for dataset creators, model providers, and venues aiming to make ``multilingual coverage'' claims verifiable rather than merely stated. Dataset: https://huggingface.co/datasets/ChialukaOnuoha/safety-slice-audit

Related papers

• The Illusion of Cross-Lingual Safety in Low-Resource Languages [7.471112976694495]
  Safety alignment in large language models (LLMs) is largely developed in English, assuming these safeguards generalize across multilingual settings.<n>We investigate cross-lingual safety transfer in four African languages, Twi, Hausa, Amharic, and Swahili, using LoDNA, a new safety dataset that pairs literal translations with culturally localized prompts.<n>Our experimental results show that cross-lingual safety transfer is severely limited; harmful prompts retain less than 10% of the English refusal signal across most language-model pairs.
  arXiv  Detail & Related papers  (2026-08-11T17:05:26Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Lower-Resource, Higher Scores: Language Bias in LLM Evaluators [27.959960538188753]
  We conduct experiments with semantically identical instruction-response pairs across 23 languages.<n>We find that multilingual evaluators assign significantly different scores to different evaluation languages.
  arXiv  Detail & Related papers  (2026-07-16T01:53:09Z)
• Sycophancy as a Multilingual Alignment Failure: How Safety Degrades Across Languages, Topics, and Models [3.2555924024638667]
  Safety-aligned large language models often exhibit sycophancy, which is the tendency to affirm users' opinions regardless of factual accuracy.<n>We present the first large-scale, multi-model evaluation of cross-lingual sycophancy, benchmarking textbfsix instruction-tuned models across textbf38 languages and textbf33 topic categories.
  arXiv  Detail & Related papers  (2026-06-07T04:50:40Z)
• Cross-Lingual Stability of LLM Judges Under Controlled Generation: Evidence from Finno-Ugric Languages [0.22009842278462158]
  Cross-lingual evaluation of large language models (LLMs) typically conflates two sources of variance: genuine model performance differences and measurement instability.<n>We investigate evaluation reliability by holding generation conditions constant while varying target language.<n>Our findings suggest that zero-shot judge transfer is unreliable for discourse-level assessment in morphologically rich languages.
  arXiv  Detail & Related papers  (2026-02-02T16:27:32Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• Is It Good Data for Multilingual Instruction Tuning or Just Bad Multilingual Evaluation for Large Language Models? [17.011882550422452]
  It is unknown whether the nature of the instruction data has an impact on the model output. It is questionable whether translated test sets can capture such nuances. We show that native or generation benchmarks reveal a notable difference between native and translated instruction data.
  arXiv  Detail & Related papers  (2024-06-18T17:43:47Z)
• Quantifying the Dialect Gap and its Correlates Across Languages [69.18461982439031]
  This work will lay the foundation for furthering the field of dialectal NLP by laying out evident disparities and identifying possible pathways for addressing them through mindful data collection.
  arXiv  Detail & Related papers  (2023-10-23T17:42:01Z)
• NusaWrites: Constructing High-Quality Corpora for Underrepresented and Extremely Low-Resource Languages [54.808217147579036]
  We conduct a case study on Indonesian local languages. We compare the effectiveness of online scraping, human translation, and paragraph writing by native speakers in constructing datasets. Our findings demonstrate that datasets generated through paragraph writing by native speakers exhibit superior quality in terms of lexical diversity and cultural content.
  arXiv  Detail & Related papers  (2023-09-19T14:42:33Z)
• No Language Left Behind: Scaling Human-Centered Machine Translation [69.28110770760506]
  We create datasets and models aimed at narrowing the performance gap between low and high-resource languages. We propose multiple architectural and training improvements to counteract overfitting while training on thousands of tasks. Our model achieves an improvement of 44% BLEU relative to the previous state-of-the-art.
  arXiv  Detail & Related papers  (2022-07-11T07:33:36Z)
• Harnessing Multilinguality in Unsupervised Machine Translation for Rare Languages [48.28540903568198]
  We show that multilinguality is critical to making unsupervised systems practical for low-resource settings. We present a single model for 5 low-resource languages (Gujarati, Kazakh, Nepali, Sinhala, and Turkish) to and from English directions. We outperform all current state-of-the-art unsupervised baselines for these languages, achieving gains of up to 14.4 BLEU.
  arXiv  Detail & Related papers  (2020-09-23T15:07:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.