Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles

• URL: http://arxiv.org/abs/2604.21152v1
• Date: Wed, 22 Apr 2026 23:33:18 GMT
• Title: Dialect vs Demographics: Quantifying LLM Bias from Implicit Linguistic Signals vs. Explicit User Profiles
• Abstract summary: Explicit identity prompts activate aggressive safety filters, increasing refusal rates and reducing semantic similarity compared to reference text for Black users.<n>In contrast, implicit dialect cues trigger a powerful dialect jailbreak,'' reducing refusal probability to near zero.<n>We find that current safety alignment techniques are brittle and over-indexed on explicit keywords, creating a bifurcated user experience.
• Score: 0.42970700836450476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As state-of-the-art Large Language Models (LLMs) have become ubiquitous, ensuring equitable performance across diverse demographics is critical. However, it remains unclear whether these disparities arise from the explicitly stated identity itself or from the way identity is signaled. In real-world interactions, users' identity is often conveyed implicitly through a complex combination of various socio-linguistic factors. This study disentangles these signals by employing a factorial design with over 24,000 responses from two open-weight LLMs (Gemma-3-12B and Qwen-3-VL-8B), comparing prompts with explicitly announced user profiles against implicit dialect signals (e.g., AAVE, Singlish) across various sensitive domains. Our results uncover a unique paradox in LLM safety where users achieve ``better'' performance by sounding like a demographic than by stating they belong to it. Explicit identity prompts activate aggressive safety filters, increasing refusal rates and reducing semantic similarity compared to our reference text for Black users. In contrast, implicit dialect cues trigger a powerful ``dialect jailbreak,'' reducing refusal probability to near zero while simultaneously achieving a greater level of semantic similarity to the reference texts compared to Standard American English prompts. However, this ``dialect jailbreak'' introduces a critical safety trade-off regarding content sanitization. We find that current safety alignment techniques are brittle and over-indexed on explicit keywords, creating a bifurcated user experience where ``standard'' users receive cautious, sanitized information while dialect speakers navigate a less sanitized, more raw, and potentially a more hostile information landscape and highlights a fundamental tension in alignment--between equitable and linguistic diversity--and underscores the need for safety mechanisms that generalize beyond explicit cues.

Related papers

• It's How You Ask: Gender-Associated Linguistic Bias in LLMs [3.2242349194283353]
  We show that when prompts contain linguistic features more commonly used by women, they elicit shorter, less sophisticated, and less formal responses.<n>Our work calls for upstream consideration of the influences of linguistic variation to mitigate disparate impacts of LLM-mediated workplace communication.
  arXiv  Detail & Related papers  (2026-08-13T14:54:23Z)
• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• SpeechJBB: Probing Safety Alignment and Comprehension in Large Audio Language Models under Code-Switched Speech [13.52804010671621]
  SpeechJBB is an audio jailbreak dataset for benchmarking across state-of-the-art LALMs.<n>Code-switched harmful audio yields substantially high jailbreak success rates (JSR), with non-English monolingual and non-English code-switched pairs exhibiting the highest attack success.<n>Pseudo-word insertion further reduces refusal rates, which demonstrates that natural-sounding obfuscation can effectively bypass safety policies.
  arXiv  Detail & Related papers  (2026-06-04T11:31:38Z)
• LASA: Language-Agnostic Semantic Alignment at the Semantic Bottleneck for LLM Safety [88.98698230989186]
  Large language models (LLMs) often demonstrate strong safety performance in high-resource languages, yet exhibit severe vulnerabilities in low-resource languages.<n>We attribute this gap to a mismatch between language-agnostic semantic understanding ability and language-dominant safety alignment biased toward high-resource languages.<n>We propose Language-Agnostic Semantic Alignment (LASA), which anchors safety alignment directly in semantic bottlenecks.
  arXiv  Detail & Related papers  (2026-04-13T15:59:50Z)
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation [58.01855677487771]
  We show that multilingual large language models (MLLMs) frequently exhibit inconsistent behaviour when the prompt's language changes.<n>We propose Cross-lingual Cultural Consistent Preference optimisation (C-3PO), a consensus-driven alignment framework.<n>C-3PO achieves up to a 0.10-point absolute increase in $_S$ over unaligned models, outperforming strong prompting and representation steering baselines.
  arXiv  Detail & Related papers  (2026-04-02T14:04:06Z)
• More Than Sum of Its Parts: Deciphering Intent Shifts in Multimodal Hate Speech Detection [49.07221734365168]
  Hate speech on social media is critical for securing cyberspace, yet relies heavily on automated detection systems.<n>We characterize semantic intent shifts where modalities interact to construct implicit hate from benign cues or neutralize toxicity through semantic inversion.<n>We propose the Asymmetric Reasoning via Courtroom Agent DEbate framework to effectively decipher these complex cues.
  arXiv  Detail & Related papers  (2026-03-22T15:46:23Z)
• MiJaBench: Revealing Minority Biases in Large Language Models via Hate Speech Jailbreaking [0.0]
  MiJaBench is an adversarial benchmark comprising 44,000 prompts across 16 minority groups.<n>Defense rates fluctuate by up to 33% within the same model solely based on the target group.<n>We release all datasets and scripts to encourage research into granular demographic alignment at GitHub.
  arXiv  Detail & Related papers  (2026-01-07T20:53:18Z)
• Learning to Extract Context for Context-Aware LLM Inference [60.376872353918394]
  User prompts to large language models (LLMs) are often ambiguous or under-specified.<n> contextual cues shaped by user intentions, prior knowledge, and risk factors influence what constitutes an appropriate response.<n>We propose a framework that extracts and leverages such contextual information from the user prompt itself.
  arXiv  Detail & Related papers  (2025-12-12T19:10:08Z)
• Say It Differently: Linguistic Styles as Jailbreak Vectors [0.763334557068953]
  We study how linguistic styles such as fear or curiosity can reframe harmful intent and elicit unsafe responses from aligned models.<n>We construct style-augmented jailbreak benchmark by transforming prompts from 3 standard datasets into 11 distinct linguistic styles.<n>Styles such as fearful, curious and compassionate are most effective and contextualized rewrites outperform templated variants.
  arXiv  Detail & Related papers  (2025-11-13T17:24:38Z)
• Uncovering the Fragility of Trustworthy LLMs through Chinese Textual Ambiguity [16.065963688326242]
  We study the trustworthiness of large language models (LLMs) when encountering ambiguous narrative text in Chinese.<n>We created a benchmark dataset by collecting and generating ambiguous sentences with context and their corresponding disambiguated pairs.<n>We discovered significant fragility in LLMs when handling ambiguity, revealing behavior that differs substantially from humans.
  arXiv  Detail & Related papers  (2025-07-30T21:50:19Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• A Generative Approach to LLM Harmfulness Mitigation with Red Flag Tokens [26.119521867045616]
  We propose augmenting the model's vocabulary with a special red flag token.<n>We train the model to insert this token whenever harmful content is generated or imminent.<n>This approach is complementary to existing safety technique.
  arXiv  Detail & Related papers  (2025-02-22T21:48:48Z)
• Who Speaks Matters: Analysing the Influence of the Speaker's Ethnicity on Hate Classification [7.014381169870851]
  Large Language Models (LLMs) offer a lucrative promise for scalable content moderation, including hate speech detection.<n>They are also known to be brittle and biased against marginalised communities and dialects.<n>This requires their applications to high-stakes tasks like hate speech detection to be critically scrutinized.
  arXiv  Detail & Related papers  (2024-10-27T16:06:24Z)
• Multilingual Jailbreak Challenges in Large Language Models [96.74878032417054]
  In this study, we reveal the presence of multilingual jailbreak challenges within large language models (LLMs) We consider two potential risky scenarios: unintentional and intentional. We propose a novel textscSelf-Defense framework that automatically generates multilingual training data for safety fine-tuning.
  arXiv  Detail & Related papers  (2023-10-10T09:44:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.