Using LLMs to Evaluate Architecture Documents: Results from a Digital Marketplace Environment

• URL: http://arxiv.org/abs/2601.19693v1
• Date: Tue, 27 Jan 2026 15:11:01 GMT
• Title: Using LLMs to Evaluate Architecture Documents: Results from a Digital Marketplace Environment
• Abstract summary: We investigate how an LLM-supported evaluation of architecture documents can support software architects to improve such artefacts.<n>We used different LLMs to analyze the quality of architecture documents and compared the results with evaluations from software architects.<n>While using LLMs in this architecture task is promising, our results showed inconsistencies that need further analyses.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI plays an increasing role during software engineering activities to make them, e.g., more efficient or provide better quality. However, it is often unclear how much benefit LLMs really provide. We concentrate on software architects and investigated how an LLM-supported evaluation of architecture documents can support software architects to improve such artefacts. In the context of a research project where a digital marketplace is developed and digital solutions should be analyzed, we used different LLMs to analyze the quality of architecture documents and compared the results with evaluations from software architects. We found out that the quality of the artifact has a strong influence on the quality of the LLM, i.e., the better the quality of the architecture document was, the more consistent were the LLM-based evaluation and the human expert evaluation. While using LLMs in this architecture task is promising, our results showed inconsistencies that need further analyses before generalizing them.

Related papers

• Towards Supporting Quality Architecture Evaluation with LLM Tools [0.48563015521171976]
  This paper investigates MS Copilot as an LLM tool to analyze quality-attribute scenarios suggested by students and reviewed by experienced architects.<n>Our initial findings reveal that the LLM produces in most cases better and more accurate results regarding risks, sensitivity points and tradeoff analysis of the quality scenarios generated manually, as well as it significantly reduces the effort required for the task.
  arXiv  Detail & Related papers  (2026-03-30T18:44:24Z)
• LLM-as-a-Judge for Software Engineering: Literature Review, Vision, and the Road Ahead [27.124885915455426]
  This paper aims to steer the community toward advancing LLM-as-a-Judge for evaluating software artifacts.<n>We envision these frameworks as reliable, robust, and scalable human surrogates capable of consistent, multi-faceted artifact evaluation by 2030.
  arXiv  Detail & Related papers  (2025-10-28T12:44:54Z)
• Expert Preference-based Evaluation of Automated Related Work Generation [54.29459509574242]
  We propose GREP, a multi-turn evaluation framework that integrates classical related work evaluation criteria with expert-specific preferences.<n>For better accessibility, we design two variants of GREP: a more precise variant with proprietary LLMs as evaluators, and a cheaper alternative with open-weight LLMs.
  arXiv  Detail & Related papers  (2025-08-11T13:08:07Z)
• Supporting architecture evaluation for ATAM scenarios with LLMs [0.6136099525128398]
  This work studies MS Copilot as an LLM tool to analyze quality scenarios suggested by students in a software architecture course.<n>Our initial study reveals that the LLM produces in most cases better and more accurate results regarding the risks, sensitivity points and tradeoff analysis of the quality scenarios.
  arXiv  Detail & Related papers  (2025-05-30T18:42:12Z)
• Software Architecture Meets LLMs: A Systematic Literature Review [4.28281840272851]
  We present a systematic literature review on the use of Large Language Models in software architecture.<n>Our findings show that while LLMs are increasingly applied to a variety of software architecture tasks, some areas, such as generating source code from architectural design, remain underexplored.
  arXiv  Detail & Related papers  (2025-05-22T14:00:29Z)
• From Code to Courtroom: LLMs as the New Software Judges [29.77858458399232]
  Large Language Models (LLMs) have been increasingly used to automate software engineering tasks such as code generation and summarization.<n>Human evaluation, while effective, is very costly and time-consuming.<n>The LLM-as-a-Judge paradigm, which employs LLMs for automated evaluation, has emerged.
  arXiv  Detail & Related papers  (2025-03-04T03:48:23Z)
• Assessing LLMs for Front-end Software Architecture Knowledge [0.0]
  Large Language Models (LLMs) have demonstrated significant promise in automating software development tasks.<n>This study investigates the capabilities of an LLM in understanding, reproducing, and generating structures within the VIPER architecture.<n> Experimental results, using ChatGPT 4 Turbo 2024-04-09, reveal that the LLM excelled in higher-order tasks like evaluating and creating, but faced challenges with lower-order tasks requiring precise retrieval of architectural details.
  arXiv  Detail & Related papers  (2025-02-26T19:33:35Z)
• Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation [55.21013307734612]
  AoPS-Instruct is a dataset of more than 600,000 high-quality QA pairs.<n>LiveAoPSBench is an evolving evaluation set with timestamps, derived from the latest forum data.<n>Our work presents a scalable approach to creating and maintaining large-scale, high-quality datasets for advanced math reasoning.
  arXiv  Detail & Related papers  (2025-01-24T06:39:38Z)
• Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity [61.54815512469125]
  This survey addresses the crucial issue of factuality in Large Language Models (LLMs) As LLMs find applications across diverse domains, the reliability and accuracy of their outputs become vital.
  arXiv  Detail & Related papers  (2023-10-11T14:18:03Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [109.8527403904657]
  We show that large language models (LLMs) possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We propose a simple method to dynamically utilize supporting documents with our judgement strategy.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)
• A Survey on Evaluation of Large Language Models [87.60417393701331]
  Large language models (LLMs) are gaining increasing popularity in both academia and industry. This paper focuses on three key dimensions: what to evaluate, where to evaluate, and how to evaluate.
  arXiv  Detail & Related papers  (2023-07-06T16:28:35Z)
• Sentiment Analysis in the Era of Large Language Models: A Reality Check [69.97942065617664]
  This paper investigates the capabilities of large language models (LLMs) in performing various sentiment analysis tasks. We evaluate performance across 13 tasks on 26 datasets and compare the results against small language models (SLMs) trained on domain-specific datasets.
  arXiv  Detail & Related papers  (2023-05-24T10:45:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.