Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation

• URL: http://arxiv.org/abs/2601.19941v1
• Date: Fri, 16 Jan 2026 20:52:42 GMT
• Title: Bench4HLS: End-to-End Evaluation of LLMs in High-Level Synthesis Code Generation
• Authors: M Zafir Sadik Khan, Kimia Azar, Hadi Kamali,
• Abstract summary: Large language models (LLMs) have shown strong capabilities in code generation, including hardware design at register-transfer level (RTL)<n>The ratio of HLS to RTL-focused studies has shifted from 1:10 to 2:10 in the past six months.<n>This growing trend highlights the need for a comprehensive benchmarking and evaluation framework dedicated to LLM-based HLS.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In last two years, large language models (LLMs) have shown strong capabilities in code generation, including hardware design at register-transfer level (RTL). While their use in high-level synthesis (HLS) remains comparatively less mature, the ratio of HLS- to RTL-focused studies has shifted from 1:10 to 2:10 in the past six months, indicating growing interest in leveraging LLMs for high-level design entry while relying on downstream synthesis for optimization. This growing trend highlights the need for a comprehensive benchmarking and evaluation framework dedicated to LLM-based HLS. To address this, We present Bench4HLS for evaluating LLM-generated HLS designs. Bench4HLS comprises 170 manually drafted and validated case studies, spanning small kernels to complex accelerators, curated from widely used public repositories. The framework supports fully automated assessment of compilation success, functional correctness via simulation, and synthesis feasibility/optimization. Crucially, Bench4HLS integrates a pluggable API for power, performance, and area (PPA) analysis across various HLS toolchains and architectures, demonstrated here with Xilinx Vitis HLS and validated on Catapult HLS. By providing a structured, extensible, and plug-and-play testbed, Bench4HLS establishes a foundational methodology for benchmarking LLMs in HLS workflows.

Related papers

• RAVEL: Reasoning Agents for Validating and Evaluating LLM Text Synthesis [78.32151470154422]
  We introduce RAVEL, an agentic framework that enables the testers to autonomously plan and execute typical synthesis operations.<n>We present C3EBench, a benchmark comprising 1,258 samples derived from professional human writings.<n>By augmenting RAVEL with SOTA LLMs as operators, we find that such agentic text synthesis is dominated by the LLM's reasoning capability.
  arXiv  Detail & Related papers  (2026-02-28T14:47:34Z)
• Evaluating and Achieving Controllable Code Completion in Code LLM [89.64782747840225]
  We present the first instruction-guided code completion benchmark, Controllable Code Completion Benchmark (C3-Bench)<n>We reveal substantial gaps in instruction-following capabilities between open-source and advanced proprietary models during code completion tasks.<n>The resulting model, Qwen2.5-Coder-C3, achieves state-of-the-art performance on C3-Bench.
  arXiv  Detail & Related papers  (2026-01-22T11:40:04Z)
• TimelyHLS: LLM-Based Timing-Aware and Architecture-Specific FPGA HLS Optimization [0.0]
  TimelyHLS generates HLS code annotated with both timing-critical and design-specific pragmas.<n>TimelyHLS consistently achieves timing closure and functional correctness across platforms.
  arXiv  Detail & Related papers  (2025-07-23T22:08:15Z)
• ForgeHLS: A Large-Scale, Open-Source Dataset for High-Level Synthesis [13.87691887333415]
  We introduce ForgeHLS, a large-scale, open-source dataset explicitly designed for machine learning (ML)-driven HLS research.<n> ForgeHLS comprises over 400k diverse designs generated from 846 kernels covering a broad range of application domains.<n>Compared to existing datasets, ForgeHLS significantly enhances scale, diversity, and design coverage.
  arXiv  Detail & Related papers  (2025-07-04T02:23:46Z)
• HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Tasks [4.71707720395444]
  We introduce HLS-Eval, the first complete benchmark and evaluation framework for HLS-driven design.<n>The benchmark includes 94 unique designs drawn from standard HLS benchmarks and novel sources.<n>Beyond the benchmark, HLS-Eval offers a modular Python framework for automated, parallel evaluation of both local and hosted LLMs.
  arXiv  Detail & Related papers  (2025-04-16T17:30:36Z)
• Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis [14.458529723566379]
  Large language models (LLMs) can be employed for programming languages such as Python and C++.<n>This paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design.
  arXiv  Detail & Related papers  (2025-02-19T17:53:59Z)
• LLM2: Let Large Language Models Harness System 2 Reasoning [65.89293674479907]
  Large language models (LLMs) have exhibited impressive capabilities across a myriad of tasks, yet they occasionally yield undesirable outputs.<n>We introduce LLM2, a novel framework that combines an LLM with a process-based verifier.<n>LLMs2 is responsible for generating plausible candidates, while the verifier provides timely process-based feedback to distinguish desirable and undesirable outputs.
  arXiv  Detail & Related papers  (2024-12-29T06:32:36Z)
• Are LLMs Any Good for High-Level Synthesis? [1.3927943269211591]
  Large Language Models (LLMs) can streamline or replace the High-Level Synthesis (HLS) process. LLMs can understand natural language specifications and translate C code or natural language specifications. This study aims to illuminate the role of LLMs in HLS, identifying promising directions for optimized hardware design in applications such as AI acceleration, embedded systems, and high-performance computing.
  arXiv  Detail & Related papers  (2024-08-19T21:40:28Z)
• An Embarrassingly Simple Approach for LLM with Strong ASR Capacity [56.30595787061546]
  We focus on solving one of the most important tasks in the field of speech processing, with speech foundation encoders and large language models (LLM) Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task.
  arXiv  Detail & Related papers  (2024-02-13T23:25:04Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning [70.38817963253034]
  This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution. We provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings.
  arXiv  Detail & Related papers  (2023-09-01T09:40:36Z)
• Zero-Shot Cross-Lingual Summarization via Large Language Models [108.30673793281987]
  Cross-lingual summarization ( CLS) generates a summary in a different target language. Recent emergence of Large Language Models (LLMs) has attracted wide attention from the computational linguistics community. In this report, we empirically use various prompts to guide LLMs to perform zero-shot CLS from different paradigms.
  arXiv  Detail & Related papers  (2023-02-28T01:27:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.