Evaluation of Adversarial Robustness in Arabic Language Models

• URL: http://arxiv.org/abs/2607.25814v1
• Date: Tue, 28 Jul 2026 14:58:28 GMT
• Title: Evaluation of Adversarial Robustness in Arabic Language Models
• Abstract summary: This study aims to assess the robustness of five state-of-the-art Arabic Language Models under a distinct set of Arabic adversarial attacks.<n> insertion of diacritics can reduce the accuracy of some models by 92% while maintaining a low distance.<n>For word-level attacks, manipulating Arabic conjunctions preserves high semantic similarity scores, low distance, and leads to an accuracy degradation of up to 58%.<n>For sentence-level attacks, paraphrasing proves its effectiveness by an average reduction of 76% in the victim models' performance.
• Score: 2.1665689529884697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emergence of the recent outstanding capabilities of Arabic Language Models has opened doors for exposing their vulnerabilities. One of the major security risks associated with such Natural Language Processing models is adversarial attacks. These attacks can deceive the model into the wrong prediction, raising critical model security and safety concerns. This study aims to assess the robustness of five state-of-the-art Arabic Language Models under a distinct set of Arabic adversarial attacks applied at various levels of granularity and using different example generation strategies. We also explore a defense technique based on adversarial training to enhance model robustness. The results show that insertion of diacritics can reduce the accuracy of some models by 92% while maintaining a low perturbation distance. For word-level attacks, manipulating Arabic conjunctions preserves high semantic similarity scores, low perturbation distance, and leads to an accuracy degradation of up to 58%. For sentence-level attacks, paraphrasing proves its effectiveness by an average reduction of 76% in the victim models' performance. While adversarial training improves overall resilience, with MARBERT being the most robust and AraBERT showing the greatest relative gains, challenges persist, particularly against character-level noise. These findings highlight both the potential and limitations of current defense strategies in morphologically rich languages like Arabic.

Related papers

• Threats to Arabic Handwriting Recognition: Investigating Black-Box Adversarial Attacks on embedded ConvNet models [0.2519906683279152]
  This study demonstrates the vulnerability of high-performing models to adversarial black-box attacks.<n>Experiments were conducted on two benchmark AHR datasets containing Arabic handwritten Characters.
  arXiv  Detail & Related papers  (2026-05-18T08:45:16Z)
• LIBERO-Plus: In-depth Robustness Analysis of Vision-Language-Action Models [49.92148175114169]
  We perform a systematic vulnerability analysis by introducing controlled perturbations across seven dimensions.<n>Models exhibit extreme sensitivity to perturbation factors, including camera viewpoints and robot initial states.<n>Surprisingly, models are largely insensitive to language variations, with further experiments revealing that models tend to ignore language instructions completely.
  arXiv  Detail & Related papers  (2025-10-15T14:51:36Z)
• Differential Robustness in Transformer Language Models: Empirical Evaluation Under Adversarial Text Attacks [3.3772986620114387]
  RoBERTa-Base and FlanT5 demonstrated remarkable resilience, maintaining accuracy even when subjected to sophisticated attacks.<n>BERT-Base showed considerable vulnerability, with TextFooler achieving a 93.75% success rate in reducing model accuracy from 48% to just 3%.
  arXiv  Detail & Related papers  (2025-09-05T21:43:06Z)
• Adversarial Training for Defense Against Label Poisoning Attacks [53.893792844055106]
  Label poisoning attacks pose significant risks to machine learning models.<n>We propose a novel adversarial training defense strategy based on support vector machines (SVMs) to counter these threats.<n>Our approach accommodates various model architectures and employs a projected gradient descent algorithm with kernel SVMs for adversarial training.
  arXiv  Detail & Related papers  (2025-02-24T13:03:19Z)
• Turning Logic Against Itself : Probing Model Defenses Through Contrastive Questions [50.40122190627256]
  We introduce POATE, a novel jailbreak technique that harnesses contrastive reasoning to provoke unethical responses.<n>PoATE crafts semantically opposing intents and integrates them with adversarial templates, steering models toward harmful outputs with remarkable subtlety.<n>To counter this, we propose Intent-Aware CoT and Reverse Thinking CoT, which decompose queries to detect malicious intent and reason in reverse to evaluate and reject harmful responses.
  arXiv  Detail & Related papers  (2025-01-03T15:40:03Z)
• Scaling Trends in Language Model Robustness [7.725206196110384]
  We study language model robustness across several classification tasks, model families, and adversarial attacks.<n>We find that in the absence of explicit safety training, larger models are not consistently more robust.<n>We find that while attack scaling outpaces adversarial training across all models studied, larger adversarially trained models might give defense the advantage in the long run.
  arXiv  Detail & Related papers  (2024-07-25T17:26:41Z)
• Certified Robustness Against Natural Language Attacks by Causal Intervention [61.62348826831147]
  Causal Intervention by Semantic Smoothing (CISS) is a novel framework towards robustness against natural language attacks. CISS is provably robust against word substitution attacks, as well as empirically robust even when perturbations are strengthened by unknown attack algorithms.
  arXiv  Detail & Related papers  (2022-05-24T19:20:48Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• Evaluating Deception Detection Model Robustness To Linguistic Variation [10.131671217810581]
  We propose an analysis of model robustness against linguistic variation in the setting of deceptive news detection. We consider two prediction tasks and compare three state-of-the-art embeddings to highlight consistent trends in model performance. We find that character or mixed ensemble models are the most effective defenses and that character perturbation-based attack tactics are more successful.
  arXiv  Detail & Related papers  (2021-04-23T17:25:38Z)
• Learning to Attack: Towards Textual Adversarial Attacking in Real-world Situations [81.82518920087175]
  Adversarial attacking aims to fool deep neural networks with adversarial examples. We propose a reinforcement learning based attack model, which can learn from attack history and launch attacks more efficiently.
  arXiv  Detail & Related papers  (2020-09-19T09:12:24Z)
• Defense for Black-box Attacks on Anti-spoofing Models by Self-Supervised Learning [71.17774313301753]
  We explore the robustness of self-supervised learned high-level representations by using them in the defense against adversarial attacks. Experimental results on the ASVspoof 2019 dataset demonstrate that high-level representations extracted by Mockingjay can prevent the transferability of adversarial examples.
  arXiv  Detail & Related papers  (2020-06-05T03:03:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.