QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture

• URL: http://arxiv.org/abs/2510.22087v1
• Date: Fri, 24 Oct 2025 23:54:17 GMT
• Title: QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture
• Abstract summary: QuArch is the first benchmark designed to facilitate the development and evaluation of large language model (LLM) capabilities in computer architecture.<n>Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking.
• Score: 36.842856470579726
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The field of computer architecture, which bridges high-level software abstractions and low-level hardware implementations, remains absent from current large language model (LLM) evaluations. To this end, we present QuArch (pronounced 'quark'), the first benchmark designed to facilitate the development and evaluation of LLM knowledge and reasoning capabilities specifically in computer architecture. QuArch provides a comprehensive collection of 2,671 expert-validated question-answer (QA) pairs covering various aspects of computer architecture, including processor design, memory systems, and interconnection networks. Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking in computer architecture. Frontier model accuracies vary widely (from 34% to 72%) on these advanced questions, highlighting persistent gaps in architectural reasoning across analysis, design, and implementation QAs. By holistically assessing fundamental skills, QuArch provides a foundation for building and measuring LLM capabilities that can accelerate innovation in computing systems. With over 140 contributors from 40 institutions, this benchmark represents a community effort to set the standard for architectural reasoning in LLM evaluation.

Related papers

• SAKE: Software Architectural Knowledge Evaluation Benchmark for Large Language Models [0.0]
  Large Language Models (LLMs) are increasingly used as assistants across the software development lifecycle.<n> Architectural decision-making depends on quality attribute trade-offs, design patterns, and system-level constraints.<n>We introduce SAKE, a standardized and reproducible benchmark for assessing software architectural knowledge.
  arXiv  Detail & Related papers  (2026-06-28T17:31:14Z)
• CodegenBench: Can LLMs Write Efficient Code Across Architectures? [17.06172900876354]
  Large language models (LLMs) have been extensively evaluated on code generation tasks for general-purpose programming and GPU-accelerated environments.<n>CodegenBench is a benchmark suite designed to evaluate the generation of efficient parallel code across three distinct hardware platforms.
  arXiv  Detail & Related papers  (2026-06-01T12:55:10Z)
• OpenCompass: A Universal Evaluation Platform for Large Language Models [62.59671563145442]
  General-purpose large language models (LLMs) have become a critical link in advancing technological development.<n>Mainstream benchmark datasets face challenges such as the diversity of task types, inconsistent evaluation criteria, and fragmentation of data and processing.<n>This paper proposes and open-source Open LLMs, a one-stop, scalable, and high-concurrency general-purpose evaluation platform.
  arXiv  Detail & Related papers  (2026-05-19T02:50:11Z)
• ArchBench: Benchmarking Generative-AI for Software Architecture Tasks [0.0]
  ArchBench is the first unified platform for benchmarking large language models on software architecture tasks.<n>It provides a command-line tool with a standardized pipeline for dataset download, inference with trajectory logging, and automated evaluation.
  arXiv  Detail & Related papers  (2026-03-18T15:26:46Z)
• Enhancing LLM-Based Neural Network Generation: Few-Shot Prompting and Efficient Validation for Automated Architecture Design [39.32515161383424]
  Large Language Models (LLMs) present a promising alternative to computationally intensive Neural Architecture Search (NAS)<n>This work introduces and validates two key contributions for computer vision.
  arXiv  Detail & Related papers  (2025-12-30T10:01:55Z)
• QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code [52.66657751895655]
  Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation.<n>This paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation.<n>We propose a self-evolving prompt optimization method that enables LLMs to evolve their internal prompt strategies.
  arXiv  Detail & Related papers  (2025-11-03T03:20:26Z)
• ArchISMiner: A Framework for Automatic Mining of Architectural Issue-Solution Pairs from Online Developer Communities [6.581259125900377]
  This study introduces ArchISMiner, a framework for mining architectural knowledge from Stack Overflow (SO)<n>ArchPI trains and evaluates multiple models, including conventional ML/DL models, Pre-trained Language Models (PLMs), and Large Language Models (LLMs)<n>ArchISPE employs an indirect supervised approach that leverages diverse features, including BERT embeddings and local TextCNN features, to extract architectural issue-solution pairs.
  arXiv  Detail & Related papers  (2025-10-24T18:46:17Z)
• LM-Searcher: Cross-domain Neural Architecture Search with LLMs via Unified Numerical Encoding [55.5535016040221]
  LM-Searcher is a novel framework for cross-domain neural architecture optimization.<n>Central to our approach is NCode, a universal numerical string representation for neural architectures.<n>Our dataset, encompassing a wide range of architecture-performance pairs, encourages robust and transferable learning.
  arXiv  Detail & Related papers  (2025-09-06T09:26:39Z)
• MAAD: Automate Software Architecture Design through Knowledge-Driven Multi-Agent Collaboration [20.14573932063689]
  We propose MAAD (Multi-Agent Architecture Design), an automated framework that employs a knowledge-driven Multi-Agent System (MAS) for architecture design.<n>MaAD orchestrates four specialized agents (i.e., Analyst, Modeler, Designer and Evaluator) to collaboratively interpret requirements specifications and produce architectural blueprints.<n>Our results show that MAAD's superiority lies in generating comprehensive architectural components and delivering insightful and structured architecture evaluation reports.
  arXiv  Detail & Related papers  (2025-07-28T23:18:25Z)
• BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models [50.17907898478795]
  We introduce BinMetric, a benchmark designed to evaluate the performance of large language models on binary analysis tasks.<n>BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks.<n>Our empirical study on this benchmark investigates the binary analysis capabilities of various state-of-the-art LLMs, revealing their strengths and limitations in this field.
  arXiv  Detail & Related papers  (2025-05-12T08:54:07Z)
• Assessing LLMs for Front-end Software Architecture Knowledge [0.0]
  Large Language Models (LLMs) have demonstrated significant promise in automating software development tasks.<n>This study investigates the capabilities of an LLM in understanding, reproducing, and generating structures within the VIPER architecture.<n> Experimental results, using ChatGPT 4 Turbo 2024-04-09, reveal that the LLM excelled in higher-order tasks like evaluating and creating, but faced challenges with lower-order tasks requiring precise retrieval of architectural details.
  arXiv  Detail & Related papers  (2025-02-26T19:33:35Z)
• A quantitative framework for evaluating architectural patterns in ML systems [49.1574468325115]
  This study proposes a framework for quantitative assessment of architectural patterns in ML systems.<n>We focus on scalability and performance metrics for cost-effective CPU-based inference.
  arXiv  Detail & Related papers  (2025-01-20T15:30:09Z)
• QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture [8.683579353811531]
  QuArch is a dataset of 1500 human-validated question pairs designed to evaluate and enhance language models' understanding of computer architecture.<n>Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%.<n>Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research.
  arXiv  Detail & Related papers  (2025-01-03T16:55:53Z)
• Enhancing Architecture Frameworks by Including Modern Stakeholders and their Views/Viewpoints [48.87872564630711]
  The stakeholders with data science and Machine Learning related concerns, such as data scientists and data engineers, are yet to be included in existing architecture frameworks.<n>We surveyed 61 subject matter experts from over 25 organizations in 10 countries.
  arXiv  Detail & Related papers  (2023-08-09T21:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.