Adversarial Prompting Framework for AI Safety Assessment

• URL: http://arxiv.org/abs/2607.13453v1
• Date: Wed, 15 Jul 2026 05:26:43 GMT
• Title: Adversarial Prompting Framework for AI Safety Assessment
• Abstract summary: This paper presents the implementation of an Adversarial Prompting Framework (APF) for a comprehensive assessment of AI safety.<n>The framework systematically evaluates the resilience of the AI model through the generation of structured adversarial prompts.<n>Results indicate significant variations in the model vulnerabilities across different attack vectors.
• Score: 0.7619404259039283
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Artificial Intelligence (AI), especially Generative AI (GenAI), adoption has increased in industries significantly in recent years. However, the use of these models may also expose systems to new forms of cyberattacks by different malicious actors -- adversarial prompt attack (APA) being one of the most prominent examples of such threats. This paper presents the implementation of an Adversarial Prompting Framework (APF) for a comprehensive assessment of AI safety. The framework systematically evaluates the resilience of the AI model through the generation of structured adversarial prompts at multiple sophistication levels, from direct harmful requests to advanced encoding-based attacks. Our implementation demonstrates the practical application of this methodology in enterprise environments, providing automated testing capabilities with quantitative security assessment metrics. The results indicate significant variations in the model vulnerabilities across different attack vectors, with encoded prompts presenting the highest success rates in bypassing safety mechanisms.

Related papers

• RIFT-Bench: Dynamic Red-teaming For Agentic AI Systems [4.789897887600273]
  We introduce RIFT-Bench, a graph representation-driven methodology for dynamic red-teaming.<n>RIFT-Bench operates in two automated phases: Discovery, which extracts system structure, and Scanning, which deploys adaptive adversarial attacks.<n>We demonstrate the effectiveness of the proposed evaluation pipeline across 45 agentic systems spanning a diverse range of implementations.
  arXiv  Detail & Related papers  (2026-06-22T20:46:56Z)
• STRIDE-AI: A Threat Modeling Framework for Generative AI Security Assessment [0.0]
  We present textitSTRIDE-AI, a framework that bridges the gap between high-level risk standards (NIST AI RMF) and technical vulnerability.<n>The framework defines a six-phase assessment lifecycle, introduces a threat modeling adaptation of classical STRIDE for AI systems, and is operationalized through a purpose-built web tool.
  arXiv  Detail & Related papers  (2026-05-16T21:27:23Z)
• DecodingTrust-Agent Platform (DTap): A Controllable and Interactive Red-Teaming Platform for AI Agents [121.77550256034]
  DecodingTrust-Agent Platform (DTap) is a controllable and interactive red-teaming platform for AI agents.<n>DTap-Red is the first autonomous red-teaming agent that explores diverse injection vectors and autonomously discovers effective attack strategies.<n>Through DTap, we conduct large-scale evaluations of popular AI agents built on various backbone models.
  arXiv  Detail & Related papers  (2026-05-06T11:59:48Z)
• Toward Risk Thresholds for AI-Enabled Cyber Threats: Enhancing Decision-Making Under Uncertainty with Bayesian Networks [0.3151064009829256]
  We propose a structured approach to developing and evaluating AI cyber risk thresholds.<n>First, we analyze existing industry cyber thresholds and identify common threshold elements.<n>Second, we propose the use of Bayesian networks as a tool for modeling AI-enabled cyber risk.
  arXiv  Detail & Related papers  (2026-01-23T23:23:12Z)
• Analyzing Code Injection Attacks on LLM-based Multi-Agent Systems in Software Development [11.76638109321532]
  We propose an architecture of a multi-agent system for the implementation phase of the software engineering process.<n>We demonstrate that while such systems can generate code quite accurately, they are vulnerable to attacks, including code injection.
  arXiv  Detail & Related papers  (2025-12-26T01:08:43Z)
• Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse [50.87630846876635]
  We develop nine detailed cyber risk models.<n>Each model decomposes attacks into steps using the MITRE ATT&CK framework.<n>Individual estimates are aggregated through Monte Carlo simulation.
  arXiv  Detail & Related papers  (2025-12-09T17:54:17Z)
• A Systematic Survey of Model Extraction Attacks and Defenses: State-of-the-Art and Perspectives [65.3369988566853]
  Recent studies have demonstrated that adversaries can replicate a target model's functionality.<n>Model Extraction Attacks pose threats to intellectual property, privacy, and system security.<n>We propose a novel taxonomy that classifies MEAs according to attack mechanisms, defense approaches, and computing environments.
  arXiv  Detail & Related papers  (2025-08-20T19:49:59Z)
• Offensive Security for AI Systems: Concepts, Practices, and Applications [0.0]
  Traditional defensive measures often fall short against the unique and evolving threats facing AI-driven technologies.<n>This paper emphasizes proactive threat simulation and adversarial testing to uncover vulnerabilities throughout the AI lifecycle.
  arXiv  Detail & Related papers  (2025-05-09T18:58:56Z)
• Position: Mind the Gap-the Growing Disconnect Between Established Vulnerability Disclosure and AI Security [56.219994752894294]
  We argue that adapting existing processes for AI security reporting is doomed to fail due to fundamental shortcomings for the distinctive characteristics of AI systems.<n>Based on our proposal to address these shortcomings, we discuss an approach to AI security reporting and how the new AI paradigm, AI agents, will further reinforce the need for specialized AI security incident reporting advancements.
  arXiv  Detail & Related papers  (2024-12-19T13:50:26Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• When Authentication Is Not Enough: On the Security of Behavioral-Based Driver Authentication Systems [53.2306792009435]
  We develop two lightweight driver authentication systems based on Random Forest and Recurrent Neural Network architectures. We are the first to propose attacks against these systems by developing two novel evasion attacks, SMARTCAN and GANCAN. Through our contributions, we aid practitioners in safely adopting these systems, help reduce car thefts, and enhance driver security.
  arXiv  Detail & Related papers  (2023-06-09T14:33:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.