LM4HPC: Towards Effective Language Model Application in High-Performance Computing

• URL: http://arxiv.org/abs/2306.14979v1
• Date: Mon, 26 Jun 2023 18:05:03 GMT
• Title: LM4HPC: Towards Effective Language Model Application in High-Performance Computing
• Authors: Le Chen and Pei-Hung Lin and Tristan Vanderbruggen and Chunhua Liao and Murali Emani and Bronis de Supinski
• Abstract summary: We design the LM4 HPC framework to facilitate the research and development of HPC software analyses and optimizations using LMs. Our framework is built on top of a range of components from different levels of the machine learning software stack, with Hugging Face-compatible APIs. The results show that LM4 HPC can help users quickly evaluate a set of state-of-the-art models and generate insightful leaderboards.
• Score: 0.46180371154032884
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, language models (LMs), such as GPT-4, have been widely used in multiple domains, including natural language processing, visualization, and so on. However, applying them for analyzing and optimizing high-performance computing (HPC) software is still challenging due to the lack of HPC-specific support. In this paper, we design the LM4HPC framework to facilitate the research and development of HPC software analyses and optimizations using LMs. Tailored for supporting HPC datasets, AI models, and pipelines, our framework is built on top of a range of components from different levels of the machine learning software stack, with Hugging Face-compatible APIs. Using three representative tasks, we evaluated the prototype of our framework. The results show that LM4HPC can help users quickly evaluate a set of state-of-the-art models and generate insightful leaderboards.

Related papers

• Exploring Code Language Models for Automated HLS-based Hardware Generation: Benchmark, Infrastructure and Analysis [49.998130983414924]
  Large language models (LLMs) can be employed for programming languages such as Python and C++. This paper explores leveraging LLMs to generate High-Level Synthesis (HLS)-based hardware design.
  arXiv  Detail & Related papers  (2025-02-19T17:53:59Z)
• LLM4AD: A Platform for Algorithm Design with Large Language Model [26.199201190378215]
  We introduce LLM4AD, a unified Python platform for algorithm design (AD) with large language models (LLMs) The platform integrates numerous key methods and supports a wide range of algorithm design tasks across various domains including optimization, machine learning, and scientific discovery. We have compiled a comprehensive suite of support resources, including tutorials, examples, a user manual, online resources, and a dedicated graphical user interface (GUI)
  arXiv  Detail & Related papers  (2024-12-23T05:12:54Z)
• A Survey of Small Language Models [104.80308007044634]
  Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources. We present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques.
  arXiv  Detail & Related papers  (2024-10-25T23:52:28Z)
• Language-Driven Engineering An Interdisciplinary Software Development Paradigm [51.29189754953934]
  Our illustration includes seven graphical Integrated Modeling Environments (IMEs) that support full code generation. Four browser-based applications that were modeled and then fully automatically generated and produced using DIME. Our technology is open source and the products presented are currently in use.
  arXiv  Detail & Related papers  (2024-02-16T13:37:57Z)
• The Landscape and Challenges of HPC Research and LLMs [12.57518012358534]
  Large language models (LLMs) have revolutionized the field of deep learning. encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks.
  arXiv  Detail & Related papers  (2024-02-03T04:21:07Z)
• MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks [5.125171374181664]
  A growing trend in AI for software development is to develop large language models (LLMs) to address a variety of programming tasks. Even LLMs applied to tasks from the high-performance computing ( HPC) domain are huge in size and demand expensive compute resources for training. This is partly because LLMs for HPC tasks are obtained by finetuning existing LLMs that support several natural and/or programming languages. We build an HPC-specific LM, named MonoCoder, which is orders of magnitude smaller than existing LMs but delivers better performance on non- HPC and HPC codes.
  arXiv  Detail & Related papers  (2023-12-20T15:11:06Z)
• HPC-GPT: Integrating Large Language Model for High-Performance Computing [3.8078849170829407]
  We propose HPC-GPT, a novel LLaMA-based model that has been supervised fine-tuning using generated QA (Question-Answer) instances for the HPC domain. To evaluate its effectiveness, we concentrate on two HPC tasks: managing AI models and datasets for HPC, and data race detection. Our experiments on open-source benchmarks yield extensive results, underscoring HPC-GPT's potential to bridge the performance gap between LLMs and HPC-specific tasks.
  arXiv  Detail & Related papers  (2023-10-03T01:34:55Z)
• Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data? [49.688233418425995]
  Struc-Bench is a comprehensive benchmark featuring prominent Large Language Models (LLMs) We propose two innovative metrics, P-Score (Prompting Score) and H-Score (Heuristical Score) Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains.
  arXiv  Detail & Related papers  (2023-09-16T11:31:58Z)
• HPC-Coder: Modeling Parallel Programs using Large Language Models [2.3101915391170573]
  We show how large language models can be applied to tasks specific to high performance and scientific codes. We introduce a new dataset of HPC and scientific codes and use it to fine-tune several pre-trained models. In our experiments, we show that this model can auto-complete HPC functions where generic models cannot.
  arXiv  Detail & Related papers  (2023-06-29T19:44:55Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• Quantitatively Assessing the Benefits of Model-driven Development in Agent-based Modeling and Simulation [80.49040344355431]
  This paper compares the use of MDD and ABMS platforms in terms of effort and developer mistakes. The obtained results show that MDD4ABMS requires less effort to develop simulations with similar (sometimes better) design quality than NetLogo.
  arXiv  Detail & Related papers  (2020-06-15T23:29:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.