OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflection

• URL: http://arxiv.org/abs/2407.16237v1
• Date: Tue, 23 Jul 2024 07:22:25 GMT
• Title: OriGen:Enhancing RTL Code Generation with Code-to-Code Augmentation and Self-Reflection
• Authors: Fan Cui, Chenyang Yin, Kexing Zhou, Youwei Xiao, Guangyu Sun, Qiang Xu, Qipeng Guo, Demin Song, Dahua Lin, Xingcheng Zhang, Yun, Liang,
• Abstract summary: OriGen is a fully open-source framework featuring self-reflection capabilities and a dataset augmentation methodology. We show that OriGen remarkably outperforms other open-source alternatives in RTL code generation.
• Score: 54.775409528658486
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Recent studies have illuminated that Large Language Models (LLMs) exhibit substantial potential in the realm of RTL (Register Transfer Level) code generation, with notable advancements evidenced by commercial models such as GPT-4 and Claude3-Opus. Despite their proficiency, these commercial LLMs often raise concerns regarding privacy and security. Conversely, open-source LLMs, which offer solutions to these concerns, have inferior performance in RTL code generation tasks to commercial models due to the lack of highquality open-source RTL datasets. To address this issue, we introduce OriGen, a fully open-source framework featuring self-reflection capabilities and a dataset augmentation methodology for generating high-quality, large-scale RTL code. We propose a novel code-to-code augmentation methodology that leverages knowledge distillation to enhance the quality of the open-source RTL code datasets. Additionally, OriGen is capable of correcting syntactic errors by leveraging a self-reflection process based on feedback from the compiler. The self-reflection ability of the model is facilitated by a carefully constructed dataset, which comprises a comprehensive collection of samples. Experimental results demonstrate that OriGen remarkably outperforms other open-source alternatives in RTL code generation, surpassing the previous best-performing LLM by 9.8% on the VerilogEval-Human benchmark. Furthermore, OriGen exhibits superior capabilities in self-reflection and error rectification, surpassing GPT-4 by 18.1% on the benchmark designed to evaluate the capability of self-reflection.

Related papers

• ITERTL: An Iterative Framework for Fine-tuning LLMs for RTL Code Generation [9.409062607311528]
  Large language models (LLMs) have demonstrated excellent performance in understanding human instructions and generating code. We introduce a simple yet effective iterative training paradigm named ITERTL. We show the model trained through our proposed approach can compete with and even outperform the state-of-the-art (SOTA) open-source model.
  arXiv  Detail & Related papers  (2024-06-28T01:44:57Z)
• Improving Logits-based Detector without Logits from Black-box LLMs [56.234109491884126]
  Large Language Models (LLMs) have revolutionized text generation, producing outputs that closely mimic human writing. We present Distribution-Aligned LLMs Detection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations.
  arXiv  Detail & Related papers  (2024-06-07T19:38:05Z)
• AICoderEval: Improving AI Domain Code Generation of Large Language Models [10.060988050644076]
  We open-source the AICoderEval dataset to facilitate research in this area. We propose CoderGen, an agent-based framework, to help LLMs generate codes related to real-world tasks. We train a more powerful task-specific code generation model, named AICoder, which is refined on llama-3 based on AICoderEval.
  arXiv  Detail & Related papers  (2024-06-07T07:45:38Z)
• AlchemistCoder: Harmonizing and Eliciting Code Capability by Hindsight Tuning on Multi-source Data [64.69872638349922]
  We present AlchemistCoder, a series of Code LLMs with enhanced code generation and generalization capabilities fine-tuned on multi-source data. We propose incorporating the data construction process into the fine-tuning data as code comprehension tasks, including instruction evolution, data filtering, and code review.
  arXiv  Detail & Related papers  (2024-05-29T16:57:33Z)
• An Empirical Study of Automated Vulnerability Localization with Large Language Models [21.84971967029474]
  Large Language Models (LLMs) have shown potential in various domains, yet their effectiveness in vulnerability localization remains underexplored. Our investigation encompasses 10+ leading LLMs suitable for code analysis, including ChatGPT and various open-source models. We explore the efficacy of these LLMs using 4 distinct paradigms: zero-shot learning, one-shot learning, discriminative fine-tuning, and generative fine-tuning.
  arXiv  Detail & Related papers  (2024-03-30T08:42:10Z)
• GOLD: Generalized Knowledge Distillation via Out-of-Distribution-Guided Language Data Generation [21.56082253577229]
  Gold is a task-agnostic data generation and knowledge distillation framework. It employs an iterative out-of-distribution-guided feedback mechanism for the LLM. An energy-based OOD evaluation approach is also introduced to deal with noisy generated data.
  arXiv  Detail & Related papers  (2024-03-28T18:08:22Z)
• How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
  Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback. Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities. We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
  arXiv  Detail & Related papers  (2024-02-25T20:07:13Z)
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback [58.20547418182074]
  We introduce StepCoder, a novel framework for code generation, consisting of two main components. CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks. FGO only optimize the model by masking the unexecuted code segments to provide Fine-Grained Optimization. Our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.
  arXiv  Detail & Related papers  (2024-02-02T13:14:31Z)
• LLM-Assisted Code Cleaning For Training Accurate Code Generators [53.087019724256606]
  We investigate data quality for code and find that making the code more structured and readable leads to improved code generation performance of the system. We build a novel data-cleaning pipeline that uses these principles to transform existing programs. We evaluate our approach on two challenging algorithmic code generation benchmarks and find that fine-tuning CodeLLaMa-7B improves the performance by up to 30% compared to fine-tuning on the original dataset.
  arXiv  Detail & Related papers  (2023-11-25T02:45:50Z)
• Evaluating Diverse Large Language Models for Automatic and General Bug Reproduction [12.851941377433285]
  Large language models (LLMs) have been demonstrated to be adept at natural language processing and code generation. Our proposed technique LIBRO could successfully reproduce about one-third of all bugs in the widely used Defects4J benchmark.
  arXiv  Detail & Related papers  (2023-11-08T08:42:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.