Differential Robustness in Transformer Language Models: Empirical Evaluation Under Adversarial Text Attacks

• URL: http://arxiv.org/abs/2509.09706v1
• Date: Fri, 05 Sep 2025 21:43:06 GMT
• Title: Differential Robustness in Transformer Language Models: Empirical Evaluation Under Adversarial Text Attacks
• Authors: Taniya Gidatkar, Oluwaseun Ajao, Matthew Shardlow,
• Abstract summary: RoBERTa-Base and FlanT5 demonstrated remarkable resilience, maintaining accuracy even when subjected to sophisticated attacks.<n>BERT-Base showed considerable vulnerability, with TextFooler achieving a 93.75% success rate in reducing model accuracy from 48% to just 3%.
• Score: 3.3772986620114387
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This study evaluates the resilience of large language models (LLMs) against adversarial attacks, specifically focusing on Flan-T5, BERT, and RoBERTa-Base. Using systematically designed adversarial tests through TextFooler and BERTAttack, we found significant variations in model robustness. RoBERTa-Base and FlanT5 demonstrated remarkable resilience, maintaining accuracy even when subjected to sophisticated attacks, with attack success rates of 0%. In contrast. BERT-Base showed considerable vulnerability, with TextFooler achieving a 93.75% success rate in reducing model accuracy from 48% to just 3%. Our research reveals that while certain LLMs have developed effective defensive mechanisms, these safeguards often require substantial computational resources. This study contributes to the understanding of LLM security by identifying existing strengths and weaknesses in current safeguarding approaches and proposes practical recommendations for developing more efficient and effective defensive strategies.

Related papers

• Replicating TEMPEST at Scale: Multi-Turn Adversarial Attacks Against Trillion-Parameter Frontier Models [0.0]
  This study employed the TEMPEST multi-turn attack framework to evaluate ten frontier models from eight vendors across 1,000 harmful behaviors.<n>Six models achieved 96% to 100% attack success rate (ASR), while four showed meaningful resistance, with ASR ranging from 42% to 78%.
  arXiv  Detail & Related papers  (2025-12-08T00:30:40Z)
• A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models [23.046017613121737]
  This paper introduces a proactive defense paradigm, shifting from passive post hoc detection to anticipatory mitigation strategies.<n>We demonstrate that proactive defense strategies offer up to 63% improvement over conventional methods in misinformation prevention.
  arXiv  Detail & Related papers  (2025-07-05T09:52:21Z)
• MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models [56.09354775405601]
  Model extraction attacks aim to replicate the functionality of a black-box model through query access.<n>Most existing defenses presume that attacker queries have out-of-distribution (OOD) samples, enabling them to detect and disrupt suspicious inputs.<n>We propose MISLEADER, a novel defense strategy that does not rely on OOD assumptions.
  arXiv  Detail & Related papers  (2025-06-03T01:37:09Z)
• Neural Antidote: Class-Wise Prompt Tuning for Purifying Backdoors in CLIP [51.04452017089568]
  Class-wise Backdoor Prompt Tuning (CBPT) is an efficient and effective defense mechanism that operates on text prompts to indirectly purify CLIP.<n>CBPT significantly mitigates backdoor threats while preserving model utility.
  arXiv  Detail & Related papers  (2025-02-26T16:25:15Z)
• Adversarial Training for Defense Against Label Poisoning Attacks [53.893792844055106]
  Label poisoning attacks pose significant risks to machine learning models.<n>We propose a novel adversarial training defense strategy based on support vector machines (SVMs) to counter these threats.<n>Our approach accommodates various model architectures and employs a projected gradient descent algorithm with kernel SVMs for adversarial training.
  arXiv  Detail & Related papers  (2025-02-24T13:03:19Z)
• Robustness of Large Language Models Against Adversarial Attacks [5.312946761836463]
  We present a comprehensive study on the robustness of GPT LLM family.<n>We employ two distinct evaluation methods to assess their resilience.<n>Our experiments reveal significant variations in the robustness of these models, demonstrating their varying degrees of vulnerability to both character-level and semantic-level adversarial attacks.
  arXiv  Detail & Related papers  (2024-12-22T13:21:15Z)
• Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models [92.79804303337522]
  Vision-Language Models (VLMs) may still be vulnerable to safety alignment issues.<n>We introduce MLAI, a novel jailbreak framework that leverages scenario-aware image generation for semantic alignment.<n>Extensive experiments demonstrate MLAI's significant impact, achieving attack success rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2.
  arXiv  Detail & Related papers  (2024-11-27T02:40:29Z)
• A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models [9.304845676825584]
  We propose a novel adversarial training framework that integrates multiple attack strategies and advanced machine learning techniques. Experiments conducted on real-world datasets, including CIFAR-10 and CIFAR-100, demonstrate that the proposed method significantly enhances model robustness.
  arXiv  Detail & Related papers  (2024-10-18T23:47:46Z)
• MirrorCheck: Efficient Adversarial Defense for Vision-Language Models [55.73581212134293]
  We propose a novel, yet elegantly simple approach for detecting adversarial samples in Vision-Language Models. Our method leverages Text-to-Image (T2I) models to generate images based on captions produced by target VLMs. Empirical evaluations conducted on different datasets validate the efficacy of our approach.
  arXiv  Detail & Related papers  (2024-06-13T15:55:04Z)
• Assessing Adversarial Robustness of Large Language Models: An Empirical Study [24.271839264950387]
  Large Language Models (LLMs) have revolutionized natural language processing, but their robustness against adversarial attacks remains a critical concern. We present a novel white-box style attack approach that exposes vulnerabilities in leading open-source LLMs, including Llama, OPT, and T5.
  arXiv  Detail & Related papers  (2024-05-04T22:00:28Z)
• FLIP: A Provable Defense Framework for Backdoor Mitigation in Federated Learning [66.56240101249803]
  We study how hardening benign clients can affect the global model (and the malicious clients) We propose a trigger reverse engineering based defense and show that our method can achieve improvement with guarantee robustness. Our results on eight competing SOTA defense methods show the empirical superiority of our method on both single-shot and continuous FL backdoor attacks.
  arXiv  Detail & Related papers  (2022-10-23T22:24:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.