DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects

• URL: http://arxiv.org/abs/2604.05318v1
• Date: Tue, 07 Apr 2026 01:43:48 GMT
• Title: DIA-HARM: Dialectal Disparities in Harmful Content Detection Across 50 English Dialects
• Abstract summary: We present DIA-HARM, the first benchmark for evaluating disinformation detection across 50 English dialects.<n>Human-written dialectal content degrades detection by 1.4-3.6% F1, while AI-generated content remains stable.<n>We release the DIA-HARM framework, D3 corpus, and evaluation tools.
• Score: 6.107850985025956
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Harmful content detectors-particularly disinformation classifiers-are predominantly developed and evaluated on Standard American English (SAE), leaving their robustness to dialectal variation unexplored. We present DIA-HARM, the first benchmark for evaluating disinformation detection robustness across 50 English dialects spanning U.S., British, African, Caribbean, and Asia-Pacific varieties. Using Multi-VALUE's linguistically grounded transformations, we introduce D3 (Dialectal Disinformation Detection), a corpus of 195K samples derived from established disinformation benchmarks. Our evaluation of 16 detection models reveals systematic vulnerabilities: human-written dialectal content degrades detection by 1.4-3.6% F1, while AI-generated content remains stable. Fine-tuned transformers substantially outperform zero-shot LLMs (96.6% vs. 78.3% best-case F1), with some models exhibiting catastrophic failures exceeding 33% degradation on mixed content. Cross-dialectal transfer analysis across 2,450 dialect pairs shows that multilingual models (mDeBERTa: 97.2% average F1) generalize effectively, while monolingual models like RoBERTa and XLM-RoBERTa fail on dialectal inputs. These findings demonstrate that current disinformation detectors may systematically disadvantage hundreds of millions of non-SAE speakers worldwide. We release the DIA-HARM framework, D3 corpus, and evaluation tools: https://github.com/jsl5710/dia-harm

Related papers

• How Robust Are LLMs to Vietnamese Dialects? [7.401730363695013]
  We present the first systematic evaluation of Large Language Models (LLMs) evaluating Vietnamese to Vietnamese dialect variation across multiple tasks.<n>VialectBench contains 400 Standard Vietnamese source instances and 2,400 human-written dialectal rewrites spanning emotion recognition (ER), natural language inference (NLI), question answering (QA), and multiple-choice question answering (MCQA)<n>Across ten instruction-tuned models, dialectal inputs reduce average performance by 2.82%, and no evaluated model is fully dialect-invariant.
  arXiv  Detail & Related papers  (2026-08-11T03:02:57Z)
• M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models [0.0]
  M-GATE is a benchmark of linguistic proficiency spanning 30 typologically diverse languages.<n>We evaluate over 50 models in more than 80 configurations.
  arXiv  Detail & Related papers  (2026-08-04T15:16:05Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Evaluating Commercial AI Chatbots as News Intermediaries [85.32040752972836]
  Best systems achieve over 90% multiple-choice accuracy on questions about events reported hours earlier.<n>Every model achieves its lowest accuracy on Hindi.<n> retrieval, not reasoning, failures drive over 70% of all errors.
  arXiv  Detail & Related papers  (2026-05-21T17:42:07Z)
• Benchmarking Bengali Dialectal Bias: A Multi-Stage Framework Integrating RAG-Based Translation and Human-Augmented RLAIF [0.3227658251731014]
  Large language models (LLMs) frequently exhibit performance biases against regional dialects of low-resource languages.<n>We propose a two-phase framework to evaluate dialectal bias in LLM question-answering across nine Bengali dialects.
  arXiv  Detail & Related papers  (2026-03-22T18:44:57Z)
• DialectGen: Benchmarking and Improving Dialect Robustness in Multimodal Generation [111.94720088481614]
  Can multimodal generative models effectively produce content given dialectal textual input?<n>We construct a new large-scale benchmark spanning six common English dialects.<n>We design a general encoder-based mitigation strategy for multimodal generative models.
  arXiv  Detail & Related papers  (2025-10-16T17:56:55Z)
• GemDetox at TextDetox CLEF 2025: Enhancing a Massively Multilingual Model for Text Detoxification on Low-resource Languages [32.22353317193898]
  We describe our submission to the PAN 2025 Multilingual Text Detoxification Challenge.<n>We apply parameter-efficient LoRA SFT fine-tuning and prompting techniques like few-shot and Chain-of-Thought.<n>Our system ranks first on high-resource and low-resource languages.
  arXiv  Detail & Related papers  (2025-09-24T10:06:40Z)
• Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models [4.66584517664999]
  We introduce a trilingual dataset of 10,193 tweets in English, Urdu, and Spanish, collected via keyword filtering.<n>Our approach, integrating attention layers with GPT-3.5 Turbo and Qwen 2.5 72B, achieves strong performance.<n>Our framework offers a robust solution for multilingual hate speech detection, fostering safer digital communities worldwide.
  arXiv  Detail & Related papers  (2025-06-09T18:53:56Z)
• "Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generation [90.09260023184932]
  Retrieval-Augmented Generation (RAG) grounds Large Language Model (LLM) output by leveraging external knowledge sources to reduce factual hallucinations. NoMIRACL is a human-annotated dataset for evaluating LLM robustness in RAG across 18 typologically diverse languages. We measure relevance assessment using: (i) hallucination rate, measuring model tendency to hallucinate, when the answer is not present in passages in the non-relevant subset, and (ii) error rate, measuring model inaccuracy to recognize relevant passages in the relevant subset.
  arXiv  Detail & Related papers  (2023-12-18T17:18:04Z)
• DISCO: A Large Scale Human Annotated Corpus for Disfluency Correction in Indo-European Languages [68.66827612799577]
  Disfluency correction (DC) is the process of removing disfluent elements like fillers, repetitions and corrections from spoken utterances to create readable and interpretable text. We present a high-quality human-annotated DC corpus covering four important Indo-European languages: English, Hindi, German and French. We show that DC leads to 5.65 points increase in BLEU scores on average when used in conjunction with a state-of-the-art Machine Translation (MT) system.
  arXiv  Detail & Related papers  (2023-10-25T16:32:02Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• Adversarial Training For Low-Resource Disfluency Correction [50.51901599433536]
  We propose an adversarially-trained sequence-tagging model for Disfluency Correction (DC) We show the benefit of our proposed technique, which crucially depends on synthetically generated disfluent data, by evaluating it for DC in three Indian languages. Our technique also performs well in removing stuttering disfluencies in ASR transcripts introduced by speech impairments.
  arXiv  Detail & Related papers  (2023-06-10T08:58:53Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• Leveraging neural representations for facilitating access to untranscribed speech from endangered languages [10.61744395262441]
  We use data selected from 7 Australian Aboriginal languages and a regional variety of Dutch. We find that representations from the middle layers of the wav2vec 2.0 Transformer offer large gains in task performance. While features extracted using the pre-trained English model yielded improved detection on all the evaluation languages, better detection performance was associated with the evaluation language's phonological similarity to English.
  arXiv  Detail & Related papers  (2021-03-26T16:44:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.