HPC-GPT: Integrating Large Language Model for High-Performance Computing
- URL: http://arxiv.org/abs/2311.12833v1
- Date: Tue, 3 Oct 2023 01:34:55 GMT
- Title: HPC-GPT: Integrating Large Language Model for High-Performance Computing
- Authors: Xianzhong Ding, Le Chen, Murali Emani, Chunhua Liao, Pei-Hung Lin,
Tristan Vanderbruggen, Zhen Xie, Alberto E. Cerpa, Wan Du
- Abstract summary: We propose HPC-GPT, a novel LLaMA-based model that has been supervised fine-tuning using generated QA (Question-Answer) instances for the HPC domain.
To evaluate its effectiveness, we concentrate on two HPC tasks: managing AI models and datasets for HPC, and data race detection.
Our experiments on open-source benchmarks yield extensive results, underscoring HPC-GPT's potential to bridge the performance gap between LLMs and HPC-specific tasks.
- Score: 3.8078849170829407
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large Language Models (LLMs), including the LLaMA model, have exhibited their
efficacy across various general-domain natural language processing (NLP) tasks.
However, their performance in high-performance computing (HPC) domain tasks has
been less than optimal due to the specialized expertise required to interpret
the model responses. In response to this challenge, we propose HPC-GPT, a novel
LLaMA-based model that has been supervised fine-tuning using generated QA
(Question-Answer) instances for the HPC domain. To evaluate its effectiveness,
we concentrate on two HPC tasks: managing AI models and datasets for HPC, and
data race detection. By employing HPC-GPT, we demonstrate comparable
performance with existing methods on both tasks, exemplifying its excellence in
HPC-related scenarios. Our experiments on open-source benchmarks yield
extensive results, underscoring HPC-GPT's potential to bridge the performance
gap between LLMs and HPC-specific tasks. With HPC-GPT, we aim to pave the way
for LLMs to excel in HPC domains, simplifying the utilization of language
models in complex computing applications.
Related papers
- Automated Text Scoring in the Age of Generative AI for the GPU-poor [49.1574468325115]
We analyze the performance and efficiency of open-source, small-scale generative language models for automated text scoring.
Results show that GLMs can be fine-tuned to achieve adequate, though not state-of-the-art, performance.
arXiv Detail & Related papers (2024-07-02T01:17:01Z) - The Landscape and Challenges of HPC Research and LLMs [12.57518012358534]
Large language models (LLMs) have revolutionized the field of deep learning.
encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks.
arXiv Detail & Related papers (2024-02-03T04:21:07Z) - Domain-Specific Code Language Models: Unraveling the Potential for HPC
Codes and Tasks [5.250454826260407]
A growing trend in AI for software development is to develop larger language models (LLMs) to address a variety of programming tasks.
Even LLMs applied to tasks from the high-performance computing ( HPC) domain are huge in size and demand expensive compute resources for training.
We build an HPC-specific LM, named MonoCoder, that is orders of magnitude smaller than existing LMs but delivers similar, if not better performance.
arXiv Detail & Related papers (2023-12-20T15:11:06Z) - Exploring the Potential of Large Language Models in Computational Argumentation [54.85665903448207]
Large language models (LLMs) have demonstrated impressive capabilities in understanding context and generating natural language.
This work aims to embark on an assessment of LLMs, such as ChatGPT, Flan models, and LLaMA2 models, in both zero-shot and few-shot settings.
arXiv Detail & Related papers (2023-11-15T15:12:15Z) - Parameter and Computation Efficient Transfer Learning for
Vision-Language Pre-trained Models [79.34513906324727]
In this paper, we aim at parameter and efficient transfer learning (PCETL) for vision-language pre-trained models.
We propose a novel dynamic architecture skipping (DAS) approach towards effective PCETL.
arXiv Detail & Related papers (2023-09-04T09:34:33Z) - Secrets of RLHF in Large Language Models Part I: PPO [81.01936993929127]
Large language models (LLMs) have formulated a blueprint for the advancement of artificial general intelligence.
reinforcement learning with human feedback (RLHF) emerges as the pivotal technological paradigm underpinning this pursuit.
In this report, we dissect the framework of RLHF, re-evaluate the inner workings of PPO, and explore how the parts comprising PPO algorithms impact policy agent training.
arXiv Detail & Related papers (2023-07-11T01:55:24Z) - LM4HPC: Towards Effective Language Model Application in High-Performance
Computing [0.46180371154032884]
We design the LM4 HPC framework to facilitate the research and development of HPC software analyses and optimizations using LMs.
Our framework is built on top of a range of components from different levels of the machine learning software stack, with Hugging Face-compatible APIs.
The results show that LM4 HPC can help users quickly evaluate a set of state-of-the-art models and generate insightful leaderboards.
arXiv Detail & Related papers (2023-06-26T18:05:03Z) - Maximize to Explore: One Objective Function Fusing Estimation, Planning,
and Exploration [87.53543137162488]
We propose an easy-to-implement online reinforcement learning (online RL) framework called textttMEX.
textttMEX integrates estimation and planning components while balancing exploration exploitation automatically.
It can outperform baselines by a stable margin in various MuJoCo environments with sparse rewards.
arXiv Detail & Related papers (2023-05-29T17:25:26Z) - AutoML-GPT: Automatic Machine Learning with GPT [74.30699827690596]
We propose developing task-oriented prompts and automatically utilizing large language models (LLMs) to automate the training pipeline.
We present the AutoML-GPT, which employs GPT as the bridge to diverse AI models and dynamically trains models with optimized hyper parameters.
This approach achieves remarkable results in computer vision, natural language processing, and other challenging areas.
arXiv Detail & Related papers (2023-05-04T02:09:43Z) - Two-step hyperparameter optimization method: Accelerating hyperparameter
search by using a fraction of a training dataset [0.15420205433587747]
We present a two-step HPO method as a strategic solution to curbing computational demands and wait times.
We present our recent application of the two-step HPO method to the development of neural network emulators for aerosol activation.
arXiv Detail & Related papers (2023-02-08T02:38:26Z) - AI-coupled HPC Workflows [1.5469452301122175]
Introduction to AI/ML models into the traditional HPC has been an enabler of highly accurate modeling.
Various modes of integrating AI/ML models to HPC computations, resulting in diverse types of AI-coupled HPC.
arXiv Detail & Related papers (2022-08-24T19:16:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.