VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation

• URL: http://arxiv.org/abs/2606.13735v1
• Date: Thu, 11 Jun 2026 12:03:49 GMT
• Title: VHDLSuite: Unified Pipeline for LLM VHDL Generation with Data Synthesis and Evaluation
• Abstract summary: We introduce VHDLSuite, a benchmark-centered infrastructure for scalable VHDL generation evaluation.<n>First, we propose a data pipeline that automatically converts Verilog designs and their accompanying testbenches into executable VHDL benchmark instances.<n>Second, we introduce VHDLBench, a benchmark with over 200 VHDL problems with complete and validated testbenches.
• Score: 7.4655567977646795
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLM) have shown impressive capabilities in Register Transfer Level (RTL) code generation, particularly for Verilog. However, evaluating their performance with other Hardware Description Languages (HDL), especially VHDL, remains limited although its distinct language characteristics, such as stricter semantic rules, introduce evaluation considerations that differ from Verilog. This lack of coverage restricts fully understanding of how well current models generalize across hardware design languages with differing structures and semantics. To address this gap, we introduce VHDLSuite, a benchmark-centered infrastructure for scalable VHDL generation evaluation, integrating automated benchmark synthesis, executable validation, and multi-model diagnostic analysis. First, we propose a data pipeline that automatically converts Verilog designs and their accompanying testbenches into executable VHDL benchmark instances, followed by VUnit/GHDL-based validation to ensure each released task is compilable, runnable, and consistently checkable in the VHDL environment. Second, we introduce VHDLBench, a benchmark with over 200 VHDL problems with complete and validated testbenches across a wide range of complexity levels. Third, we extensively evaluate cutting-edge LLMs and uncover key challenges specific on LLM-aided VHDL generation. Our findings provide important insights and support future work in multi-language hardware design automation.Our data pipeline, benchmark, and evaluation framework will be open-sourced.

Related papers

• LocalV: Exploiting Information Locality for IP-level Verilog Generation [45.78831906080782]
  The generation of Register-Transfer Level (RTL) code is a crucial yet labor-intensive step in digital hardware design.<n>Existing approaches-including fine-tuned domain-specific models and advanced agent-based systems-struggle to scale to industrial IP-level design tasks.<n>We propose LocalV, a multi-agent framework that leverages information locality in modular hardware design.
  arXiv  Detail & Related papers  (2026-01-31T13:01:16Z)
• Chain-of-Descriptions: Improving Code LLMs for VHDL Code Generation and Summarization [4.7966941517322725]
  Large Language Models (LLMs) have become widely used across diverse NLP tasks and domains.<n>LLMs show promise for tasks like Register-Transfer Level (RTL) code generation and summarization.<n>We propose Chain-of-Descriptions (CoDes) to enhance the performance of LLMs for VHDL code generation and summarization tasks.
  arXiv  Detail & Related papers  (2025-07-16T15:05:30Z)
• hdl2v: A Code Translation Dataset for Enhanced LLM Verilog Generation [0.9149489479543916]
  We present.<n>HDL-to-Verilog, a dataset which seeks to increase the amount of available human-written Verilog data.<n>We demonstrate the value of.<n>HDL-to-Verilog by improving performance of a 32 billion-weight model by up to 23%.
  arXiv  Detail & Related papers  (2025-06-05T01:29:18Z)
• HDLxGraph: Bridging Large Language Models and HDL Repositories via HDL Graph Databases [57.51078142561683]
  Large Language Models (LLMs) have demonstrated their potential in hardware design tasks.<n>Yet, their performance in real-world, repository-level HDL projects with thousands or even tens of thousands of code lines is hindered.<n>We propose HDLxGraph, a novel framework that integrates Graph Retrieval Augmented Generation (Graph RAG) with LLMs.
  arXiv  Detail & Related papers  (2025-05-21T16:14:10Z)
• BinMetric: A Comprehensive Binary Analysis Benchmark for Large Language Models [50.17907898478795]
  We introduce BinMetric, a benchmark designed to evaluate the performance of large language models on binary analysis tasks.<n>BinMetric comprises 1,000 questions derived from 20 real-world open-source projects across 6 practical binary analysis tasks.<n>Our empirical study on this benchmark investigates the binary analysis capabilities of various state-of-the-art LLMs, revealing their strengths and limitations in this field.
  arXiv  Detail & Related papers  (2025-05-12T08:54:07Z)
• OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMs [62.68905180014956]
  We introduce OpenCodeInstruct, the largest open-access instruction tuning dataset, comprising 5 million diverse samples.<n>Each sample includes a programming question, solution, test cases, execution feedback, and LLM-generated quality assessments.<n>We fine-tune various base models, including LLaMA and Qwen, across multiple scales (1B+, 3B+, and 7B+) using our dataset.
  arXiv  Detail & Related papers  (2025-04-05T02:52:16Z)
• Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis [14.458529723566379]
  Large language models (LLMs) can be employed for programming languages such as Python and C++.<n>This paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design.
  arXiv  Detail & Related papers  (2025-02-19T17:53:59Z)
• CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarization [32.462699328256384]
  Traditional methods of adapting large language models for hardware design rely on synthetic HDL datasets.<n>We propose an efficient LLM fine-tuning pipeline for HDL generation that integrates a multi-level summarization data synthesis process with a novel Chat-FIM-Tag supervised fine-tuning method.
  arXiv  Detail & Related papers  (2024-07-15T03:57:20Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• VHDL-Eval: A Framework for Evaluating Large Language Models in VHDL Code Generation [4.700008016247411]
  This paper introduces a comprehensive evaluation framework designed specifically for assessing VHDL code generation task. This dataset is constructed by translating a collection of Verilog evaluation problems to VHDL and aggregating publicly available VHDL problems, resulting in a total of 202 problems. To assess the functional correctness of the generated VHDL code, we utilize a curated set of self-verifying testbenches.
  arXiv  Detail & Related papers  (2024-06-06T00:06:50Z)
• PPTC-R benchmark: Towards Evaluating the Robustness of Large Language Models for PowerPoint Task Completion [96.47420221442397]
  We construct adversarial user instructions by attacking user instructions at sentence, semantic, and multi-language levels. We test 3 closed-source and 4 open-source LLMs using a benchmark that incorporates robustness settings. We find that GPT-4 exhibits the highest performance and strong robustness in our benchmark.
  arXiv  Detail & Related papers  (2024-03-06T15:33:32Z)
• Vision-Language Instruction Tuning: A Review and Analysis [52.218690619616474]
  Vision-Language Instruction Tuning (VLIT) presents more complex characteristics compared to pure text instruction tuning. We offer a detailed categorization for existing VLIT datasets and identify the characteristics that high-quality VLIT data should possess. By incorporating these characteristics as guiding principles into the existing VLIT data construction process, we conduct extensive experiments and verify their positive impact on the performance of tuned multi-modal LLMs.
  arXiv  Detail & Related papers  (2023-11-14T14:02:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.