Towards Leveraging LLMs to Generate Abstract Penetration Test Cases from Software Architecture

• URL: http://arxiv.org/abs/2603.23698v1
• Date: Tue, 24 Mar 2026 20:32:18 GMT
• Title: Towards Leveraging LLMs to Generate Abstract Penetration Test Cases from Software Architecture
• Abstract summary: We propose the generation of Abstract Penetration Test Cases (APTCs) from software architecture models as an input to support architecture-level security assessment.<n>We analyze relevant standards and state of the art using two criteria: (i) derivability from software architecture, and (ii) usability for both architecture security assessment and subsequent penetration testing.<n>Our evaluation shows promising results, achieving up to 93% usefulness and 86% correctness, indicating that the generated APTCs can substantially support both architects and penetration testers.
• Score: 1.8447974135158336
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Software architecture models capture early design decisions that strongly influence system quality attributes, including security. However, architecture-level security assessment and feedback are often absent in practice, allowing security weaknesses to propagate into later phases of the software development lifecycle and, in some cases, to remain undiscovered, ultimately leading to vulnerable systems. In this paper, we bridge this gap by proposing the generation of Abstract Penetration Test Cases (APTCs) from software architecture models as an input to support architecture-level security assessment. We first introduce a metamodel that defines the APTC concept, and then investigate the use of large language models with different prompting strategies to generate meaningful APTCs from architecture models. To design the APTC metamodel, we analyze relevant standards and state of the art using two criteria: (i) derivability from software architecture, and (ii) usability for both architecture security assessment and subsequent penetration testing. Building on this metamodel, we then proceed to generate APTCs from software architecture models. Our evaluation shows promising results, achieving up to 93\% usefulness and 86\% correctness, indicating that the generated APTCs can substantially support both architects (by highlighting security-critical design decisions) and penetration testers (by providing actionable testing guidance).

Related papers

• Bridging Requirements and Architecture: Multi-Agent Orchestration with External Knowledge and Hierarchical Memory [37.071260774880884]
  We propose MAAD (Multi-Agent Architecture Design), a knowledge-driven framework that orchestrates four specialized agents to attribute architectural blueprints.<n>We show that MAAD generates more complete, modular, and traceable architectures than the baseline, and its dedicated Evaluator agent autonomously produces structured quality evaluation reports.
  arXiv  Detail & Related papers  (2026-05-31T18:18:31Z)
• Benchmarking Requirement-to-Architecture Generation with Hybrid Evaluation [8.867061253116367]
  Large Language Models (LLMs) have demonstrated significant potential in automating software engineering tasks.<n> Generating software architecture designs from requirement documents is a crucial step in software development.<n>R2ABench is a novel benchmark comprising diverse real-world software projects paired with comprehensive Product Requirements Documents (PRDs) and expert-curated PlantUML reference diagrams.
  arXiv  Detail & Related papers  (2026-04-08T04:58:36Z)
• Architecture-Aware Multi-Design Generation for Repository-Level Feature Addition [53.50448142467294]
  RAIM is a multi-design and architecture-aware framework for repository-level feature addition.<n>It shifts away from linear patching by generating multiple diverse implementation designs.<n>Experiments on the NoCode-bench Verified dataset demonstrate that RAIM establishes a new state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-03-02T12:50:40Z)
• A Benchmark for Language Models in Real-World System Building [56.549267258789904]
  Cross-ISA software package repair is a critical task for ensuring the reliability of software deployment and the stability of modern operating systems.<n>We introduce a new benchmark designed for software package build repair across diverse architectures and languages.<n>We evaluate six state-of-the-art LLMs on the benchmark, and the results show that cross-ISA software package repair remains difficult and requires further advances.
  arXiv  Detail & Related papers  (2026-01-19T10:30:46Z)
• Training Language Models to Generate Quality Code with Program Analysis Feedback [66.0854002147103]
  Code generation with large language models (LLMs) is increasingly adopted in production but fails to ensure code quality.<n>We propose REAL, a reinforcement learning framework that incentivizes LLMs to generate production-quality code.
  arXiv  Detail & Related papers  (2025-05-28T17:57:47Z)
• How Does Sequence Modeling Architecture Influence Base Capabilities of Pre-trained Language Models? Exploring Key Architecture Design Principles to Avoid Base Capabilities Degradation [47.61640436552693]
  This work focuses on the impact of sequence modeling architectures on base capabilities.<n>We first point out that the mixed domain pre-training setting fails to adequately reveal the differences in base capabilities among various architectures.<n>Next, we analyze the base capabilities of stateful sequence modeling architectures, and find that they exhibit significant degradation in base capabilities compared to the Transformer.
  arXiv  Detail & Related papers  (2025-05-24T05:40:03Z)
• ATRAF-driven IMRaD Methodology: Tradeoff and Risk Analysis of Software Architectures Across Abstraction Levels [0.0]
  evaluating architectural artifacts is essential to assess tradeoffs and risks affecting quality attributes such as performance, modifiability, and security.<n>Our prior work introduced the Architecture Tradeoff and Risk Analysis Framework (ATRAF)<n>This paper presents the ATRAF-driven IMRaD methodology, a concise method to align ATRAF's phases with IMRaD sections.
  arXiv  Detail & Related papers  (2025-05-06T15:22:28Z)
• The Architecture Tradeoff and Risk Analysis Framework (ATRAF): A Unified Approach for Evaluating Software Architectures, Reference Architectures, and Architectural Frameworks [0.0]
  We introduce the Architecture Tradeoff and Risk Analysis Framework (ATRAF)<n>ATRAF is a scenario-driven framework for evaluating tradeoffs and risks across architectural levels.<n>It enables the identification of sensitivities, tradeoffs, and risks while supporting continuous refinement of architectural artifacts.
  arXiv  Detail & Related papers  (2025-05-01T17:48:52Z)
• Thinking Longer, Not Larger: Enhancing Software Engineering Agents via Scaling Test-Time Compute [61.00662702026523]
  We propose a unified Test-Time Compute scaling framework that leverages increased inference-time instead of larger models.<n>Our framework incorporates two complementary strategies: internal TTC and external TTC.<n>We demonstrate our textbf32B model achieves a 46% issue resolution rate, surpassing significantly larger models such as DeepSeek R1 671B and OpenAI o1.
  arXiv  Detail & Related papers  (2025-03-31T07:31:32Z)
• A Survey of Model Architectures in Information Retrieval [59.61734783818073]
  The period from 2019 to the present has represented one of the biggest paradigm shifts in information retrieval (IR) and natural language processing (NLP)<n>We trace the development from traditional term-based methods to modern neural approaches, particularly highlighting the impact of transformer-based models and subsequent large language models (LLMs)<n>We conclude with a forward-looking discussion of emerging challenges and future directions.
  arXiv  Detail & Related papers  (2025-02-20T18:42:58Z)
• Model-Driven Security Analysis of Self-Sovereign Identity Systems [2.5475486924467075]
  We propose a model-driven security analysis framework for analyzing architectural patterns of SSI systems. Our framework mechanizes a modeling language to formalize patterns and threats with security properties in temporal logic. We present typical vulnerable patterns verified by SecureSSI.
  arXiv  Detail & Related papers  (2024-06-02T05:44:32Z)
• From Requirements to Architecture: An AI-Based Journey to Semi-Automatically Generate Software Architectures [2.4150871564195007]
  We propose a method to generate software architecture candidates based on requirements using artificial intelligence techniques. We further envision an automatic evaluation and trade-off analysis of the generated architecture candidates.
  arXiv  Detail & Related papers  (2024-01-25T10:56:58Z)
• Towards Responsible Generative AI: A Reference Architecture for Designing Foundation Model based Agents [28.406492378232695]
  Foundation model based agents derive their autonomy from the capabilities of foundation models. This paper presents a pattern-oriented reference architecture that serves as guidance when designing foundation model based agents.
  arXiv  Detail & Related papers  (2023-11-22T04:21:47Z)
• Serving Deep Learning Model in Relational Databases [70.53282490832189]
  Serving deep learning (DL) models on relational data has become a critical requirement across diverse commercial and scientific domains. We highlight three pivotal paradigms: The state-of-the-art DL-centric architecture offloads DL computations to dedicated DL frameworks. The potential UDF-centric architecture encapsulates one or more tensor computations into User Defined Functions (UDFs) within the relational database management system (RDBMS)
  arXiv  Detail & Related papers  (2023-10-07T06:01:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.