SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models

• URL: http://arxiv.org/abs/2605.25420v1
• Date: Mon, 25 May 2026 04:45:44 GMT
• Title: SomaliBench Eval: Measuring English-to-Somali Refusal Gaps in Open-Weight Language Models
• Abstract summary: We evaluate four open-weight instruction-tuned models on SomaliBench v0.<n>We find large English-to-Somali refusal gaps for all four models.<n>For three models, the dominant Somali non-refusal mode is not fluent harmful compliance but unclear output.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model safety evaluation remains heavily English-centered, leaving low-resource languages under-measured even when models are deployed globally. We evaluate four open-weight instruction-tuned models on SomaliBench v0, a native-author-verified benchmark of 100 harmful-intent prompts paired across English and Somali. Each of Llama-3.1-8B-Instruct, Gemma-2-9B-Instruct, Qwen-2.5-7B-Instruct, and Aya-23-8B is run locally with temperature 0 and the same English "helpful, harmless, and honest" (HHH) system prompt. A pinned Claude Sonnet snapshot (claude-sonnet-4-5-20250929) classifies each response as refused, complied, or unclear; the native author spot-checks a stratified 80-row sample. We find large English-to-Somali refusal gaps for all four models: Llama-3.1-8B (0.90; 95% bootstrap CI [0.85, 0.96]), Aya-23-8B (0.75 [0.67, 0.83]), Qwen-2.5-7B (0.69 [0.59, 0.78]), and Gemma-2-9B (0.38 [0.27, 0.49]). For three models, the dominant Somali non-refusal mode is not fluent harmful compliance but unclear output: empty, wrong-language, or incoherent generations. The native verification spot-check achieves 100% agreement with the judge (Cohen's kappa = 1.00) on the 80 sampled rows. We report aggregate refusal rates, category gaps, and reliability statistics only; raw model generations are retained locally and are not released.

Related papers

• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models [0.37331950863394864]
  We present a rubric-based benchmark for the Saudi dialect.<n>We catalogue 466 error instances under a nine-category taxonomy.<n>We observe a consistency-versus-ceiling trade-off and model-distinctive error signatures.
  arXiv  Detail & Related papers  (2026-08-30T19:31:35Z)
• Interpretable Cross-Lingual Alignment in Small Language Models: Probing Cultural and Pragmatic Reasoning in Japanese-English Bilingual LLMs [0.0]
  I introduce J-PragEval-v0, a minimal-pair benchmark isolating four phenomena from surface fluency.<n>I combine it with linear probes and teacher-forced log-probability evaluation to ask where inside TinySwallow-1.5B the corresponding contrasts live.
  arXiv  Detail & Related papers  (2026-08-14T21:04:21Z)
• Accurate Ensembles, Fragile Narratives: Multi-Scale Stacking and a Fidelity Audit of LLM-Generated Explanations for Credit Risk [0.0]
  Credit scoring increasingly relies on models whose decision logic cannot be read off their parameters.<n>A common proposal closes that gap with a language model: compute feature attributions, hand them to an LLM, and let it write the rationale.<n>We build such a system end to end and test whether the second half of the promise holds.
  arXiv  Detail & Related papers  (2026-08-08T13:22:14Z)
• A Comparative Study of Language Models for Khmer Retrieval-Augmented Question Answering [0.0]
  Retrieval-Augmented Generation (RAG) has emerged as a promising paradigm for grounding large language model (LLM) outputs in retrieved evidence.<n>We present a RAG-based question answering system for Khmer-language telecom-domain documents.
  arXiv  Detail & Related papers  (2026-05-21T07:36:01Z)
• RefusalBench: Why Refusal Rate Misranks Frontier LLMs on Biological Research Prompts [0.0]
  Strict refusal rates span 0.1% to 94.6% on identical prompts.<n>Nine of 18 frontier models exhibit a hedge-but-help partial-compliance pattern at dual-use tier that binary refusal metrics cannot detect.
  arXiv  Detail & Related papers  (2026-05-20T09:53:31Z)
• Articulate but Wrong: Self-Review Failures in LLM-Based Code Modernization [1.0164694825170502]
  Large language model (LLM) agents are increasingly used to migrate legacy code to modern stacks.<n>We run 1,980 real modernization calls across 11 production LLMs from 7 distinct families on a balanced 60-snippet legacy-Python-2 corpus.<n>We find that semantic-preservation drift is prevalent and sharply separable from a cleanly-controlled baseline.
  arXiv  Detail & Related papers  (2026-05-20T05:00:31Z)
• Benchmarking Linguistic Adaptation in Comparable-Sized LLMs: A Study of Llama-3.1-8B, Mistral-7B-v0.1, and Qwen3-8B on Romanized Nepali [0.0]
  Romanized Nepali, the Nepali language written in the Latin alphabet, is the dominant medium for informal digital communication in Nepal.<n>This study presents a systematic benchmarking of linguistic adaptation across three comparable-sized open-weight models.
  arXiv  Detail & Related papers  (2026-03-25T07:02:51Z)
• BrainBench: Exposing the Commonsense Reasoning Gap in Large Language Models [4.264192013842096]
  Large language models (LLMs) achieve impressive scores on standard benchmarks yet routinely fail questions that any human would answer correctly in seconds.<n>We introduce BrainBench, a benchmark of 100 brainteaser questions spanning 20 carefully designed categories.
  arXiv  Detail & Related papers  (2026-03-16T02:50:43Z)
• RefineBench: Evaluating Refinement Capability of Language Models via Checklists [71.02281792867531]
  We evaluate two refinement modes: guided refinement and self-refinement.<n>In guided refinement, both proprietary LMs and large open-weight LMs can leverage targeted feedback to refine responses to near-perfect levels within five turns.<n>These findings suggest that frontier LMs require breakthroughs to self-refine their incorrect responses.
  arXiv  Detail & Related papers  (2025-11-27T07:20:52Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• Paloma: A Benchmark for Evaluating Language Model Fit [112.481957296585]
  Evaluations of language models (LMs) commonly report perplexity on monolithic data held out from training.<n>We introduce Perplexity Analysis for Language Model Assessment (Paloma), a benchmark to measure LM fit to 546 English and code domains.
  arXiv  Detail & Related papers  (2023-12-16T19:12:45Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)
• SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languages [102.50127671423752]
  We introduce SMaLL-100, a distilled version of the M2M-100 (12B) machine translation model covering 100 languages. We train SMaLL-100 with uniform sampling across all language pairs and therefore focus on preserving the performance of low-resource languages. Our model achieves comparable results to M2M-100 (1.2B), while being 3.6x smaller and 4.3x faster at inference.
  arXiv  Detail & Related papers  (2022-10-20T22:32:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.