Emerging Challenges in Threat Modeling for GenAI-Augmented Systems: A View from the Trenches

• URL: http://arxiv.org/abs/2607.28431v1
• Date: Thu, 30 Jul 2026 16:08:33 GMT
• Title: Emerging Challenges in Threat Modeling for GenAI-Augmented Systems: A View from the Trenches
• Abstract summary: Threat modeling remains a central task in secure software engineering.<n>Traditional threat modeling methods are insufficient to assess emerging GenAI-specific risks.<n>We present the first results from an exploratory assessment of GenAI-aware threat modeling methods in a Small and Medium Enterprise setting.
• Score: 3.5141073261469202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Threat modeling remains a central task in secure software engineering, as it enables the identification of security issues from system architectures. As Generative Artificial Intelligence (GenAI) becomes increasingly pervasive across software systems, traditional threat modeling methods (e.g., STRIDE) are insufficient to assess emerging GenAI-specific risks. In this work, we present the first results from an exploratory assessment of GenAI-aware threat modeling methods in a Small and Medium Enterprise (SME) setting. For this, we conducted a rapid literature review to select relevant techniques and systematically applied three shortlisted methods to an industrial case study involving a GenAI-augmented system. The results highlight differences in the threats identified by each technique and reveal limited support for certain GenAI-specific risk categories, particularly those related to software supply chains and human-centered security issues. We further report practitioners' perceptions of the usability and integration of these methods in SME development workflows, including their perceived effort and adoption challenges.

Related papers

• A LINDDUN-based Privacy Threat Modeling Framework for GenAI [13.683645364910568]
  We introduce a novel domain-specific privacy threat modeling framework to support the privacy threat analysis of GenAI-based applications.<n>The new framework affects three out of the seven privacy threat types of LINDDUN and introduces 100 new GenAI examples to the knowledge base.<n>Its effectiveness is validated on an AI Agent system, which demonstrates that a comprehensive privacy analysis can be supported by the new framework.
  arXiv  Detail & Related papers  (2026-03-06T09:04:32Z)
• Frontier AI Risk Management Framework in Practice: A Risk Analysis Technical Report v1.5 [61.787178868669265]
  This technical report presents an updated and granular assessment of five critical dimensions: cyber offense, persuasion and manipulation, strategic deception, uncontrolled AI R&D, and self-replication.<n>This work reflects our current understanding of AI frontier risks and urges collective action to mitigate these challenges.
  arXiv  Detail & Related papers  (2026-02-16T04:30:06Z)
• Toward Quantitative Modeling of Cybersecurity Risks Due to AI Misuse [50.87630846876635]
  We develop nine detailed cyber risk models.<n>Each model decomposes attacks into steps using the MITRE ATT&CK framework.<n>Individual estimates are aggregated through Monte Carlo simulation.
  arXiv  Detail & Related papers  (2025-12-09T17:54:17Z)
• Generative AI for Biosciences: Emerging Threats and Roadmap to Biosecurity [56.331312963880215]
  generative artificial intelligence (GenAI) in the biosciences is transforming biotechnology, medicine, and synthetic biology.<n>This Perspective outlines the current state of GenAI in the biosciences and emerging threat vectors ranging from jailbreak attacks and privacy risks to the dual-use challenges posed by autonomous AI agents.<n>We advocate a multi-layered approach to GenAI safety, including rigorous data filtering, alignment with ethical principles during development, and real-time monitoring to block harmful requests.
  arXiv  Detail & Related papers  (2025-10-13T00:24:41Z)
• GenAI Security: Outsmarting the Bots with a Proactive Testing Framework [0.0]
  This research explores the need for proactive security measures to mitigate the risks associated with malicious exploitation of GenAI systems.<n>We present a framework encompassing key approaches, tools, and strategies designed to outmaneuver adversarial attacks.
  arXiv  Detail & Related papers  (2025-05-14T12:55:05Z)
• A Peek Behind the Curtain: Using Step-Around Prompt Engineering to Identify Bias and Misinformation in GenAI Models [0.0]
  We discuss how Internet-sourced training data introduces unintended biases and misinformation into AI systems.<n>We argue that step-around prompting serves a vital role in identifying potential vulnerabilities while acknowledging its dual nature as both a research tool and a security threat.
  arXiv  Detail & Related papers  (2025-03-19T13:47:28Z)
• Computational Safety for Generative AI: A Signal Processing Perspective [65.268245109828]
  computational safety is a mathematical framework that enables the quantitative assessment, formulation, and study of safety challenges in GenAI.<n>We show how sensitivity analysis and loss landscape analysis can be used to detect malicious prompts with jailbreak attempts.<n>We discuss key open research challenges, opportunities, and the essential role of signal processing in computational AI safety.
  arXiv  Detail & Related papers  (2025-02-18T02:26:50Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)
• Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systems [88.80306881112313]
  We will introduce and define a family of approaches to AI safety, which we will refer to as guaranteed safe (GS) AI. The core feature of these approaches is that they aim to produce AI systems which are equipped with high-assurance quantitative safety guarantees. We outline a number of approaches for creating each of these three core components, describe the main technical challenges, and suggest a number of potential solutions to them.
  arXiv  Detail & Related papers  (2024-05-10T17:38:32Z)
• Asset-centric Threat Modeling for AI-based Systems [7.696807063718328]
  This paper presents ThreatFinderAI, an approach and tool to model AI-related assets, threats, countermeasures, and quantify residual risks. To evaluate the practicality of the approach, participants were tasked to recreate a threat model developed by cybersecurity experts of an AI-based healthcare platform. Overall, the solution's usability was well-perceived and effectively supports threat identification and risk discussion.
  arXiv  Detail & Related papers  (2024-03-11T08:40:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.