SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models

• URL: http://arxiv.org/abs/2606.29520v1
• Date: Sun, 28 Jun 2026 17:31:14 GMT
• Title: SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models
• Abstract summary: Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle.<n> Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints.<n>We introduce SAKE, a standardized and reproducible benchmark for assessing software architectural knowledge.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle, yet their ability to reason about software architecture remains largely unmeasured. Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints, none of which are exercised by benchmarks that target syntactic or algorithmic tasks. We introduce SAKE (Software Architectural Knowledge Evaluation), a standardized and reproducible benchmark for assessing software architectural knowledge in LLMs. SAKE comprises 2154 expert-curated multiple-choice questions, each with four options, stratified across eight architectural categories and four context-length levels. We evaluate 11 proprietary and open-weight models in zero-shot and five-shot settings. Overall accuracy is high, but performance varies markedly across categories, revealing competency gaps in areas central to professional practice. SAKE, its evaluation scripts, and all results are released as open source to give the community a baseline for tracking architectural reasoning in LLMs.

Related papers

• Bridging Requirements and Architecture: Multi-Agent Orchestration with External Knowledge and Hierarchical Memory [37.071260774880884]
  We propose MAAD (Multi-Agent Architecture Design), a knowledge-driven framework that orchestrates four specialized agents to attribute architectural blueprints.<n>We show that MAAD generates more complete, modular, and traceable architectures than the baseline, and its dedicated Evaluator agent autonomously produces structured quality evaluation reports.
  arXiv  Detail & Related papers  (2026-05-31T18:18:31Z)
• CAKE: Cloud Architecture Knowledge Evaluation of Large Language Models [1.1140384738063094]
  CAKE consists of 188 expert-validated questions covering four cognitive levels of Bloom's revised taxonomy.<n> Evaluation is conducted on 22 model configurations using three-run majority voting for multiple-choice questions (MCQs) and LLM-as-a-judge scoring for free-responses (FR)
  arXiv  Detail & Related papers  (2026-04-07T11:56:43Z)
• SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models [59.90381306452982]
  evaluating large language models (LLMs) for software engineering has been limited by narrow task coverage, language bias, and insufficient alignment with real-world developer.<n>We introduce SWE-1, a comprehensive benchmark that unifies heterogeneous code-related evaluations into a structured and production-aligned framework.<n>SWE- spans 8 task types, 8 programming scenarios, and 10 programming languages, with 2000 high-quality instances curated from authentic GitHub pull requests.
  arXiv  Detail & Related papers  (2025-11-07T18:01:32Z)
• EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits [72.23150343093447]
  We introduce EDIT-Bench, a benchmark for evaluatingstructed code editing capabilities grounded in real-world usage.<n>EDIT-Bench comprises of 545 problems, multiple natural and programming languages, and a diverse set of real-world use cases.<n>We find that model performance varies across different categories of user instructions.
  arXiv  Detail & Related papers  (2025-11-06T16:05:28Z)
• Collaborative LLM Agents for C4 Software Architecture Design Automation [0.0]
  This study contributes to automated software architecture design and its evaluation methods.<n>We introduce an LLM-based multi-agent system that automates the production of a C4 software architecture model.<n>Tested on five canonical system briefs, the workflow demonstrates fast C4 model creation, sustains high compilation success, and delivers semantic fidelity.
  arXiv  Detail & Related papers  (2025-10-26T18:43:59Z)
• QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture [36.842856470579726]
  QuArch is the first benchmark designed to facilitate the development and evaluation of large language model (LLM) capabilities in computer architecture.<n>Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking.
  arXiv  Detail & Related papers  (2025-10-24T23:54:17Z)
• MAAD: Automate Software Architecture Design through Knowledge-Driven Multi-Agent Collaboration [20.14573932063689]
  We propose MAAD (Multi-Agent Architecture Design), an automated framework that employs a knowledge-driven Multi-Agent System (MAS) for architecture design.<n>MaAD orchestrates four specialized agents (i.e., Analyst, Modeler, Designer and Evaluator) to collaboratively interpret requirements specifications and produce architectural blueprints.<n>Our results show that MAAD's superiority lies in generating comprehensive architectural components and delivering insightful and structured architecture evaluation reports.
  arXiv  Detail & Related papers  (2025-07-28T23:18:25Z)
• MERA Code: A Unified Framework for Evaluating Code Generation Across Tasks [56.34018316319873]
  We propose MERA Code, a benchmark for evaluating code for the latest code generation LLMs in Russian.<n>This benchmark includes 11 evaluation tasks that span 8 programming languages.<n>We evaluate open LLMs and frontier API models, analyzing their limitations in terms of practical coding tasks in non-English languages.
  arXiv  Detail & Related papers  (2025-07-16T14:31:33Z)
• BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models [50.17907898478795]
  We introduce BinMetric, a benchmark designed to evaluate the performance of large language models on binary analysis tasks.<n>BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks.<n>Our empirical study on this benchmark investigates the binary analysis capabilities of various state-of-the-art LLMs, revealing their strengths and limitations in this field.
  arXiv  Detail & Related papers  (2025-05-12T08:54:07Z)
• The Digital Cybersecurity Expert: How Far Have We Come? [49.89857422097055]
  We develop CSEBenchmark, a fine-grained cybersecurity evaluation framework based on 345 knowledge points expected of cybersecurity experts.<n>We evaluate 12 popular large language models (LLMs) on CSEBenchmark and find that even the best-performing model achieves only 85.42% overall accuracy.<n>By identifying and addressing specific knowledge gaps in each LLM, we achieve up to an 84% improvement in correcting previously incorrect predictions.
  arXiv  Detail & Related papers  (2025-04-16T05:36:28Z)
• Semi-Automated Design of Data-Intensive Architectures [49.1574468325115]
  This paper introduces a development methodology for data-intensive architectures.<n>It guides architects in (i) designing a suitable architecture for their specific application scenario, and (ii) selecting an appropriate set of concrete systems to implement the application.<n>We show that the description languages we adopt can capture the key aspects of data-intensive architectures proposed by researchers and practitioners.
  arXiv  Detail & Related papers  (2025-03-21T16:01:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.