The Adaptive Arms Race: Redefining Robustness in AI Security

• URL: http://arxiv.org/abs/2312.13435v3
• Date: Tue, 06 May 2025 14:21:22 GMT
• Title: The Adaptive Arms Race: Redefining Robustness in AI Security
• Authors: Ilias Tsingenopoulos, Vera Rimmer, Davy Preuveneers, Fabio Pierazzi, Lorenzo Cavallaro, Wouter Joosen,
• Abstract summary: We introduce a framework for adaptively optimizing black-box attacks and defenses under the competitive game they form.<n>We find that active defenses, those that dynamically control system responses, are an essential complement to model hardening against decision-based attacks.<n>Our findings, supported by an extensive theoretical and empirical investigation, confirm that adaptive adversaries pose a serious threat to black-box AI-based systems.
• Score: 21.759075171536388
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite considerable efforts on making them robust, real-world AI-based systems remain vulnerable to decision based attacks, as definitive proofs of their operational robustness have so far proven intractable. Canonical robustness evaluation relies on adaptive attacks, which leverage complete knowledge of the defense and are tailored to bypass it. This work broadens the notion of adaptivity, which we employ to enhance both attacks and defenses, showing how they can benefit from mutual learning through interaction. We introduce a framework for adaptively optimizing black-box attacks and defenses under the competitive game they form. To assess robustness reliably, it is essential to evaluate against realistic and worst-case attacks. We thus enhance attacks and their evasive arsenal together using RL, apply the same principle to defenses, and evaluate them first independently and then jointly under a multi-agent perspective. We find that active defenses, those that dynamically control system responses, are an essential complement to model hardening against decision-based attacks; that these defenses can be circumvented by adaptive attacks, something that elicits defenses being adaptive too. Our findings, supported by an extensive theoretical and empirical investigation, confirm that adaptive adversaries pose a serious threat to black-box AI-based systems, rekindling the proverbial arms race. Notably, our approach outperforms the state-of-the-art black-box attacks and defenses, while bringing them together to render effective insights into the robustness of real-world deployed ML-based systems.

Related papers

• MISLEADER: Defending against Model Extraction with Ensembles of Distilled Models [56.09354775405601]
  Model extraction attacks aim to replicate the functionality of a black-box model through query access.<n>Most existing defenses presume that attacker queries have out-of-distribution (OOD) samples, enabling them to detect and disrupt suspicious inputs.<n>We propose MISLEADER, a novel defense strategy that does not rely on OOD assumptions.
  arXiv  Detail & Related papers  (2025-06-03T01:37:09Z)
• Concealment of Intent: A Game-Theoretic Analysis [15.387256204743407]
  We present a scalable attack strategy: intent-hiding adversarial prompting, which conceals malicious intent through the composition of skills.<n>Our analysis identifies equilibrium points and reveals structural advantages for the attacker.<n> Empirically, we validate the attack's effectiveness on multiple real-world LLMs across a range of malicious behaviors.
  arXiv  Detail & Related papers  (2025-05-27T07:59:56Z)
• Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents [3.5248694676821484]
  We evaluate eight different defenses and bypass all of them using adaptive attacks, consistently achieving an attack success rate of over 50%. Our research underscores the need for adaptive attack evaluation when designing defenses to ensure robustness and reliability.
  arXiv  Detail & Related papers  (2025-02-27T04:04:50Z)
• A Novel Approach to Guard from Adversarial Attacks using Stable Diffusion [0.0]
  Our proposal suggests a different approach to the AI Guardian framework. Instead of including adversarial examples in the training process, we propose training the AI system without them. This aims to create a system that is inherently resilient to a wider range of attacks.
  arXiv  Detail & Related papers  (2024-05-03T04:08:15Z)
• Position: Towards Resilience Against Adversarial Examples [42.09231029292568]
  We provide a definition of adversarial resilience and outline considerations of designing an adversarially resilient defense. We then introduce a subproblem of adversarial resilience which we call continual adaptive robustness. We demonstrate the connection between continual adaptive robustness and previously studied problems of multiattack robustness and unforeseen attack robustness.
  arXiv  Detail & Related papers  (2024-05-02T14:58:44Z)
• CARE: Ensemble Adversarial Robustness Evaluation Against Adaptive Attackers for Security Applications [14.25922051336361]
  Ensemble defenses are widely employed in various security-related applications to enhance model performance and robustness. There are no platforms for comprehensive evaluation of ensemble adversarial attacks and defenses in the cybersecurity domain.
  arXiv  Detail & Related papers  (2024-01-20T05:37:09Z)
• Avoid Adversarial Adaption in Federated Learning by Multi-Metric Investigations [55.2480439325792]
  Federated Learning (FL) facilitates decentralized machine learning model training, preserving data privacy, lowering communication costs, and boosting model performance through diversified data sources. FL faces vulnerabilities such as poisoning attacks, undermining model integrity with both untargeted performance degradation and targeted backdoor attacks. We define a new notion of strong adaptive adversaries, capable of adapting to multiple objectives simultaneously. MESAS is the first defense robust against strong adaptive adversaries, effective in real-world data scenarios, with an average overhead of just 24.37 seconds.
  arXiv  Detail & Related papers  (2023-06-06T11:44:42Z)
• A Random-patch based Defense Strategy Against Physical Attacks for Face Recognition Systems [3.6202815454709536]
  We propose a random-patch based defense strategy to robustly detect physical attacks for Face Recognition System (FRS) Our method can be easily applied to the real world face recognition system and extended to other defense methods to boost the detection performance.
  arXiv  Detail & Related papers  (2023-04-16T16:11:56Z)
• Ares: A System-Oriented Wargame Framework for Adversarial ML [3.197282271064602]
  Ares is an evaluation framework for adversarial ML that allows researchers to explore attacks and defenses in a realistic wargame-like environment. Ares frames the conflict between the attacker and defender as two agents in a reinforcement learning environment with opposing objectives. This allows the introduction of system-level evaluation metrics such as time to failure and evaluation of complex strategies.
  arXiv  Detail & Related papers  (2022-10-24T04:55:18Z)
• Scale-Invariant Adversarial Attack for Evaluating and Enhancing Adversarial Defenses [22.531976474053057]
  Projected Gradient Descent (PGD) attack has been demonstrated to be one of the most successful adversarial attacks. We propose Scale-Invariant Adversarial Attack (SI-PGD), which utilizes the angle between the features in the penultimate layer and the weights in the softmax layer to guide the generation of adversaries.
  arXiv  Detail & Related papers  (2022-01-29T08:40:53Z)
• Fixed Points in Cyber Space: Rethinking Optimal Evasion Attacks in the Age of AI-NIDS [70.60975663021952]
  We study blackbox adversarial attacks on network classifiers. We argue that attacker-defender fixed points are themselves general-sum games with complex phase transitions. We show that a continual learning approach is required to study attacker-defender dynamics.
  arXiv  Detail & Related papers  (2021-11-23T23:42:16Z)
• Model-Agnostic Meta-Attack: Towards Reliable Evaluation of Adversarial Robustness [53.094682754683255]
  We propose a Model-Agnostic Meta-Attack (MAMA) approach to discover stronger attack algorithms automatically. Our method learns the in adversarial attacks parameterized by a recurrent neural network. We develop a model-agnostic training algorithm to improve the ability of the learned when attacking unseen defenses.
  arXiv  Detail & Related papers  (2021-10-13T13:54:24Z)
• Adversarial defense for automatic speaker verification by cascaded self-supervised learning models [101.42920161993455]
  More and more malicious attackers attempt to launch adversarial attacks at automatic speaker verification (ASV) systems. We propose a standard and attack-agnostic method based on cascaded self-supervised learning models to purify the adversarial perturbations. Experimental results demonstrate that the proposed method achieves effective defense performance and can successfully counter adversarial attacks.
  arXiv  Detail & Related papers  (2021-02-14T01:56:43Z)
• Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses [59.58128343334556]
  We introduce a relaxation term to the standard loss, that finds more suitable gradient-directions, increases attack efficacy and leads to more efficient adversarial training. We propose Guided Adversarial Margin Attack (GAMA), which utilizes function mapping of the clean image to guide the generation of adversaries. We also propose Guided Adversarial Training (GAT), which achieves state-of-the-art performance amongst single-step defenses.
  arXiv  Detail & Related papers  (2020-11-30T16:39:39Z)
• On Adaptive Attacks to Adversarial Example Defenses [123.32678153377915]
  This paper lays out the methodology and the approach necessary to perform an adaptive attack against defenses to adversarial examples. We hope that these analyses will serve as guidance on how to properly perform adaptive attacks against defenses to adversarial examples.
  arXiv  Detail & Related papers  (2020-02-19T18:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.