ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety

• URL: http://arxiv.org/abs/2605.14152v1
• Date: Wed, 13 May 2026 22:07:22 GMT
• Title: ROK-FORTRESS: Measuring the Effect of Geopolitical Transcreation for National Security and Public Safety
• Abstract summary: Safety evaluations for large language models increasingly target high-stakes National Security and Public Safety (NSPS) risks.<n> empirical evidence of how language and geopolitical context interact remains limited to a narrow set of language pairs.<n>We introduce emphROK-FORTRESS, a bilingual, culturally adversarial NSPS benchmark that uses the English--Korean language pair and U.S.--ROK geopolitical axis as a case study.
• Score: 2.929429337200341
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Safety evaluations for large language models (LLMs) increasingly target high-stakes National Security and Public Safety (NSPS) risks, yet multilingual safety is typically assessed through translation-only benchmarks that preserve the underlying scenario, and empirical evidence of how language and geopolitical context interact remains limited to a narrow set of language pairs. We introduce \emph{ROK-FORTRESS} https://huggingface.co/datasets/ScaleAI/ROK-FORTRESS_public, a bilingual, culturally adversarial NSPS benchmark that uses the English--Korean language pair and U.S.--ROK geopolitical axis as a case study, separating the effects of language and geopolitical grounding via a \emph{transcreation matrix}: adversarial intents are evaluated under controlled combinations of (i) English versus Korean language and (ii) U.S.\ versus Korean entities, institutions, and operational details. Each adversarial prompt is paired with a dual-use benign counterpart to quantify over-refusal. Model responses are then scored using calibrated LLM-as-a-judge panels, applying our expert-crafted, prompt-specific binary rubrics. Across a dual-track set of frontier and Korean-optimized models, we find a consistent suppression effect in Korean variants and substantial model-to-model variation in how geopolitical grounding interacts with language. In many models, Korean grounding mitigates the Korean language-driven suppression -- with no model showing significant amplification in the other direction -- indicating that, at least in the English--Korean case, safety behavior is shaped by language-as-risk signals and context interactions that translation-only evaluations miss. The transcreation matrix methodology is designed to generalize to other language--culture pairs.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• IndicGuard: A Multilingual Safety Guard Model and Dataset for Indic Languages [2.584263027095689]
  We introduce IndicGuard, a multilingual safety guard model and dataset for Indic languages.<n>We construct a high-volume, culturally nuanced safety dataset encompassing ten major Indic languages.<n>We fine-tune a 4B- parameter instruction-tuned model based on Gemma-3-4B-IT to serve as a multilingual safety guardrail.
  arXiv  Detail & Related papers  (2026-06-22T04:36:57Z)
• Schützen: Evaluating LLM Safety in Bulgarian and German Contexts [53.865251738592605]
  This paper introduces Schtzen: a German--Bulgarian safety dataset designed to assess model answerability under risk.<n> Experiments with multilingual and language-specific LLMs reveal pronounced cross-language differences in safety behavior.
  arXiv  Detail & Related papers  (2026-06-09T18:01:19Z)
• Culturally-Adapted Red-Teaming Across East and Southeast Asian Contexts: A Methodological and Comparative Analysis [2.8523456804049885]
  We construct paired DT and culturally-adapted datasets via 1:1 seed matching for four languages.<n>We compare Attack Success Rate (ASR) and Cultural Realism scores across four open-source safety evaluations.
  arXiv  Detail & Related papers  (2026-06-08T08:17:18Z)
• Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs [0.0]
  We present the first systematic cross-lingual, multimodal red-teaming study comparing jailbreak vulnerability in US English (en-US) and Mexican Spanish (es-MX)<n>Our central finding is that language does not scale vulnerability uniformly.<n>This indicates that linguistic and visual alignment failures operate through distinct mechanisms, and that switching language is sufficient to expose that separation.
  arXiv  Detail & Related papers  (2026-05-22T02:12:45Z)
• Multilingual Refusal Alignment for Safer Large Language Models [53.64286756804503]
  We investigate whether single-language alignment transfers cross-lingually, how language consistency is preserved during training, and the resulting trade-offs with general knowledge capabilities.<n>We introduce RefusEU, a novel refusal alignment dataset covering 12 European languages, including a dedicated test set for evaluating current state-of-the-art models.<n>Our controlled Direct Preference Optimization (DPO) experiments provide two key insights: aligning models exclusively in English is insufficient to ensure cross-lingual safety, even for the same harm categories, whereas training on multilingual datasets can improve safety without degrading general performance, as measured by the Global MMLU benchmark.
  arXiv  Detail & Related papers  (2026-04-24T09:29:14Z)
• Lost in Translation? A Comparative Study on the Cross-Lingual Transfer of Composite Harms [0.5376203747548287]
  Most safety evaluations of large language models (LLMs) remain anchored in English.<n>Some types of harm survive translation almost intact, while others distort or disappear.<n>We introduce CompositeHarm, a translation-based benchmark to examine how safety alignment holds up as both syntax and semantics shift.
  arXiv  Detail & Related papers  (2026-02-08T13:22:50Z)
• When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training [57.230355403478995]
  We investigate the development of language-agnostic concept spaces during pretraining of EuroLLM.<n>We find that shared concept spaces emerge early and continue to refine, but that alignment with them is language-dependent.<n>In contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior.
  arXiv  Detail & Related papers  (2026-01-30T11:23:01Z)
• Improving Methodologies for LLM Evaluations Across Global Languages [19.63570354411416]
  The exercise shows how safety behaviours can vary across languages.<n>It also generated insights for improving multilingual safety evaluations.<n>This work represents an initial step toward a shared framework for multilingual safety testing of advanced AI systems.
  arXiv  Detail & Related papers  (2026-01-22T07:18:08Z)
• Multilinguality as Sense Adaptation [24.548610248136352]
  SENse-based Symmetric Interlingual Alignment (SENSIA)<n>We introduce SENse-based Symmetric Interlingual Alignment (SENSIA)<n>It adapts a Backpack language model from one language to another by explicitly aligning sense-level mixtures and contextual representations on parallel data.
  arXiv  Detail & Related papers  (2026-01-15T11:44:01Z)
• Toxicity Red-Teaming: Benchmarking LLM Safety in Singapore's Low-Resource Languages [57.059267233093465]
  Large Language Models (LLMs) have transformed natural language processing, but their safety mechanisms remain under-explored in low-resource, multilingual settings.<n>We introduce textsfSGToxicGuard, a novel dataset and evaluation framework for benchmarking LLM safety in Singapore's diverse linguistic context.<n>We conduct extensive experiments with state-of-the-art multilingual LLMs, and the results uncover critical gaps in their safety guardrails.
  arXiv  Detail & Related papers  (2025-09-18T08:14:34Z)
• Humans overrely on overconfident language models, across languages [32.71245803698373]
  We study the risks of multilingual linguistic (mis)calibration, overconfidence, and overreliance across five languages.<n>Our work finds that overreliance risks are high across languages.
  arXiv  Detail & Related papers  (2025-07-08T18:01:01Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• The Hidden Space of Safety: Understanding Preference-Tuned LLMs in Multilingual context [0.9130277390156759]
  Alignment tuning has enabled large language models to excel in reasoning, instruction-following, and minimizing harmful generations.<n>Despite their widespread deployment, these models exhibit a monolingual bias, raising concerns about the effectiveness of alignment across languages.<n>Current alignment methods predominantly focus on English, leaving it unclear how alignment mechanism generalizes to multilingual settings.
  arXiv  Detail & Related papers  (2025-04-03T15:46:46Z)
• High-Dimensional Interlingual Representations of Large Language Models [65.77317753001954]
  Large language models (LLMs) trained on massive multilingual datasets hint at the formation of interlingual constructs.<n>We explore 31 diverse languages varying on their resource-levels, typologies, and geographical regions.<n>We find that multilingual LLMs exhibit inconsistent cross-lingual alignments.
  arXiv  Detail & Related papers  (2025-03-14T10:39:27Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.