HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages

• URL: http://arxiv.org/abs/2412.15178v1
• Date: Thu, 19 Dec 2024 18:52:05 GMT
• Title: HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
• Authors: Aman Chaturvedi, Daniel Nichols, Siddharth Singh, Abhinav Bhatele,
• Abstract summary: Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants. They are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing. We conduct an in-depth study along the many axes of fine-tuning a specialized HPC LLM in order to better understand the challenges.
• Score: 1.6954729278440728
• License:
• Abstract: Large Language Model (LLM) based coding tools have been tremendously successful as software development assistants, yet they are often designed for general purpose programming tasks and perform poorly for more specialized domains such as high performance computing. Creating specialized models and tools for these domains is crucial towards gaining the benefits of LLMs in areas such as HPC. While previous work has explored HPC-specific models, LLMs still struggle to generate parallel code and it is not at all clear what hurdles are still holding back these LLMs and what must be done to overcome them. In this work, we conduct an in-depth study along the many axes of fine-tuning a specialized HPC LLM in order to better understand the challenges. Based on our findings we fine-tune and evaluate a specialized HPC LLM that is shown to be the best performing open-source code LLM for parallel code generation to date.

Related papers

• Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis [49.998130983414924]
  Large language models (LLMs) can be employed for programming languages such as Python and C++. This paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design.
  arXiv  Detail & Related papers  (2025-02-19T17:53:59Z)
• Bridge-Coder: Unlocking LLMs' Potential to Overcome Language Gaps in Low-Resource Code [31.48411893252137]
  Large Language Models (LLMs) demonstrate strong proficiency in generating code for high-resource programming languages (HRPLs) like Python but struggle significantly with low-resource programming languages (LRPLs) such as Racket or D. This performance gap deepens the digital divide, preventing developers using LRPLs from benefiting equally from LLM advancements and reinforcing disparities in innovation within underrepresented programming communities. We introduce a novel approach called Bridge-Coder, which leverages LLMs' intrinsic capabilities to enhance the performance on LRPLs.
  arXiv  Detail & Related papers  (2024-10-24T17:55:03Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning [53.6472920229013]
  Large Language Models (LLMs) have demonstrated impressive capability in many natural language tasks. LLMs are prone to produce errors, hallucinations and inconsistent statements when performing multi-step reasoning. We introduce Q*, a framework for guiding LLMs decoding process with deliberative planning.
  arXiv  Detail & Related papers  (2024-06-20T13:08:09Z)
• An Embarrassingly Simple Approach for LLM with Strong ASR Capacity [56.30595787061546]
  We focus on solving one of the most important tasks in the field of speech processing, with speech foundation encoders and large language models (LLM) Recent works have complex designs such as compressing the output temporally for the speech encoder, tackling modal alignment for the projector, and utilizing parameter-efficient fine-tuning for the LLM. We found that delicate designs are not necessary, while an embarrassingly simple composition of off-the-shelf speech encoder, LLM, and the only trainable linear projector is competent for the ASR task.
  arXiv  Detail & Related papers  (2024-02-13T23:25:04Z)
• StepCoder: Improve Code Generation with Reinforcement Learning from Compiler Feedback [58.20547418182074]
  We introduce StepCoder, a novel framework for code generation, consisting of two main components. CCCS addresses the exploration challenge by breaking the long sequences code generation task into a Curriculum of Code Completion Subtasks. FGO only optimize the model by masking the unexecuted code segments to provide Fine-Grained Optimization. Our method improves the ability to explore the output space and outperforms state-of-the-art approaches in corresponding benchmarks.
  arXiv  Detail & Related papers  (2024-02-02T13:14:31Z)
• If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents [81.60906807941188]
  Large language models (LLMs) are trained on a combination of natural language and formal language (code) Code translates high-level goals into executable steps, featuring standard syntax, logical consistency, abstraction, and modularity.
  arXiv  Detail & Related papers  (2024-01-01T16:51:20Z)
• MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks [5.125171374181664]
  A growing trend in AI for software development is to develop large language models (LLMs) to address a variety of programming tasks. Even LLMs applied to tasks from the high-performance computing ( HPC) domain are huge in size and demand expensive compute resources for training. This is partly because LLMs for HPC tasks are obtained by finetuning existing LLMs that support several natural and/or programming languages. We build an HPC-specific LM, named MonoCoder, which is orders of magnitude smaller than existing LMs but delivers better performance on non- HPC and HPC codes.
  arXiv  Detail & Related papers  (2023-12-20T15:11:06Z)
• A Survey of Large Language Models for Code: Evolution, Benchmarking, and Future Trends [30.774685501251817]
  General large language models (LLMs) have demonstrated significant potential in tasks such as code generation in software engineering. A considerable portion of Code LLMs is derived from general LLMs through model fine-tuning. There is currently a lack of systematic investigation into Code LLMs and their performance.
  arXiv  Detail & Related papers  (2023-11-17T07:55:16Z)
• Scope is all you need: Transforming LLMs for HPC Code [5.0227775038998415]
  We propose a novel tokenizer named Tokompiler, designed specifically for preprocessing code in HPC and compilation-centric tasks. Tokompiler leverages knowledge of language primitives to generate language-oriented tokens, providing a context-aware understanding of code structure. Results demonstrate that Tokompiler significantly enhances code completion accuracy and semantic understanding compared to traditional tokenizers.
  arXiv  Detail & Related papers  (2023-08-18T10:12:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.