QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture

• URL: http://arxiv.org/abs/2501.01892v2
• Date: Mon, 06 Jan 2025 17:48:05 GMT
• Title: QuArch: A Question-Answering Dataset for AI Agents in Computer Architecture
• Abstract summary: QuArch is a dataset of 1500 human-validated question pairs designed to evaluate and enhance language models' understanding of computer architecture.<n>Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%.<n>Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research.
• Score: 8.683579353811531
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce QuArch, a dataset of 1500 human-validated question-answer pairs designed to evaluate and enhance language models' understanding of computer architecture. The dataset covers areas including processor design, memory systems, and performance optimization. Our analysis highlights a significant performance gap: the best closed-source model achieves 84% accuracy, while the top small open-source model reaches 72%. We observe notable struggles in memory systems, interconnection networks, and benchmarking. Fine-tuning with QuArch improves small model accuracy by up to 8%, establishing a foundation for advancing AI-driven computer architecture research. The dataset and leaderboard are at https://harvard-edge.github.io/QuArch/.

Related papers

• QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture [36.842856470579726]
  QuArch is the first benchmark designed to facilitate the development and evaluation of large language model (LLM) capabilities in computer architecture.<n>Our evaluation reveals that while frontier models possess domain-specific knowledge, they struggle with skills that require higher-order thinking.
  arXiv  Detail & Related papers  (2025-10-24T23:54:17Z)
• Dynamic Acoustic Model Architecture Optimization in Training for ASR [51.21112094223223]
  DMAO is an architecture optimization framework that employs a grow-and-drop strategy to automatically reallocate parameters during training.<n>We evaluate DMAO through experiments with CTC onSpeech, TED-LIUM-v2 and Switchboard datasets.
  arXiv  Detail & Related papers  (2025-06-16T07:47:34Z)
• Knowledge-aware Evolutionary Graph Neural Architecture Search [49.13787973318586]
  Graph neural architecture search (GNAS) can customize high-performance graph neural network architectures for specific graph tasks or datasets. Existing GNAS methods begin searching for architectures from a zero-knowledge state, ignoring the prior knowledge that may improve the search efficiency. This study proposes exploiting such prior knowledge to accelerate the multi-objective evolutionary search on a new graph dataset.
  arXiv  Detail & Related papers  (2024-11-26T11:32:45Z)
• Depthwise Separable Convolutions with Deep Residual Convolutions [0.0]
  Xception is one of the most popular deep learning algorithms for computer vision applications. The computational complexity of Xception sometimes hinders its deployment on resource-constrained edge devices. We incorporate the depthwise separable convolutions with deep residual convolutions of the Xception architecture to develop a small and efficient model for edge devices.
  arXiv  Detail & Related papers  (2024-11-12T04:47:32Z)
• Building Optimal Neural Architectures using Interpretable Knowledge [15.66288233048004]
  AutoBuild is a scheme which learns to align the latent embeddings of operations and architecture modules with the ground-truth performance of the architectures they appear in. We show that by mining a relatively small set of evaluated architectures, AutoBuild can learn to build high-quality architectures directly or help to reduce search space to focus on relevant areas.
  arXiv  Detail & Related papers  (2024-03-20T04:18:38Z)
• How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources [117.6496550359768]
  This work explores recent advances in instruction-tuning language models on a range of open instruction-following datasets. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets. We evaluate them on their factual knowledge, reasoning, multilinguality, coding, and open-ended instruction following abilities.
  arXiv  Detail & Related papers  (2023-06-07T19:59:23Z)
• Learning Interpretable Models Through Multi-Objective Neural Architecture Search [0.9990687944474739]
  We propose a framework to optimize for both task performance and "introspectability," a surrogate metric for aspects of interpretability. We demonstrate that jointly optimizing for task error and introspectability leads to more disentangled and debuggable architectures that perform within error.
  arXiv  Detail & Related papers  (2021-12-16T05:50:55Z)
• Network Graph Based Neural Architecture Search [57.78724765340237]
  We search neural network by rewiring the corresponding graph and predict the architecture performance by graph properties. Because we do not perform machine learning over the entire graph space, the searching process is remarkably efficient.
  arXiv  Detail & Related papers  (2021-12-15T00:12:03Z)
• DAAS: Differentiable Architecture and Augmentation Policy Search [107.53318939844422]
  This work considers the possible coupling between neural architectures and data augmentation and proposes an effective algorithm jointly searching for them. Our approach achieves 97.91% accuracy on CIFAR-10 and 76.6% Top-1 accuracy on ImageNet dataset, showing the outstanding performance of our search algorithm.
  arXiv  Detail & Related papers  (2021-09-30T17:15:17Z)
• Neural Architecture Search using Covariance Matrix Adaptation Evolution Strategy [6.8129169853808795]
  We propose a framework of applying the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) to the neural architecture search problem called CMANAS. The architecture are modelled using a normal distribution, which is updated using CMA-ES based on the fitness of the sampled population. CMANAS finished the architecture search on CIFAR-10 with the top-1 test accuracy of 97.44% in 0.45 GPU day and on CIFAR-100 with the top-1 test accuracy of 83.24% for 0.6 GPU day on a single GPU.
  arXiv  Detail & Related papers  (2021-07-15T11:41:23Z)
• Evolving Neural Architecture Using One Shot Model [5.188825486231326]
  We propose a novel way of applying a simple genetic algorithm to the NAS problem called EvNAS (Evolving Neural Architecture using One Shot Model) EvNAS searches for the architecture on the proxy dataset i.e. CIFAR-10 for 4.4 GPU day on a single GPU and achieves top-1 test error of 2.47%. Results show the potential of evolutionary methods in solving the architecture search problem.
  arXiv  Detail & Related papers  (2020-12-23T08:40:53Z)
• A Semi-Supervised Assessor of Neural Architectures [157.76189339451565]
  We employ an auto-encoder to discover meaningful representations of neural architectures. A graph convolutional neural network is introduced to predict the performance of architectures.
  arXiv  Detail & Related papers  (2020-05-14T09:02:33Z)
• Tidying Deep Saliency Prediction Architectures [6.613005108411055]
  In this paper, we identify four key components of saliency models, i.e., input features, multi-level integration, readout architecture, and loss functions. We propose two novel end-to-end architectures called SimpleNet and MDNSal, which are neater, minimal, more interpretable and achieve state of the art performance on public saliency benchmarks.
  arXiv  Detail & Related papers  (2020-03-10T19:34:49Z)
• Semi-Supervised Neural Architecture Search [185.0651567642238]
  SemiNAS is a semi-supervised Neural architecture search (NAS) approach that leverages numerous unlabeled architectures (without evaluation and thus nearly no cost) It achieves 94.02% test accuracy on NASBench-101, outperforming all the baselines when using the same number of architectures. It achieves 97% intelligibility rate in the low-resource setting and 15% test error rate in the robustness setting, with 9%, 7% improvements over the baseline respectively.
  arXiv  Detail & Related papers  (2020-02-24T17:23:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.