LM4HPC: Towards Effective Language Model Application in High-Performance Computing

• URL: http://arxiv.org/abs/2306.14979v1
• Date: Mon, 26 Jun 2023 18:05:03 GMT
• Title: LM4HPC: Towards Effective Language Model Application in High-Performance Computing
• Authors: Le Chen and Pei-Hung Lin and Tristan Vanderbruggen and Chunhua Liao and Murali Emani and Bronis de Supinski
• Abstract summary: We design the LM4 HPC framework to facilitate the research and development of HPC software analyses and optimizations using LMs. Our framework is built on top of a range of components from different levels of the machine learning software stack, with Hugging Face-compatible APIs. The results show that LM4 HPC can help users quickly evaluate a set of state-of-the-art models and generate insightful leaderboards.
• Score: 0.46180371154032884
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, language models (LMs), such as GPT-4, have been widely used in multiple domains, including natural language processing, visualization, and so on. However, applying them for analyzing and optimizing high-performance computing (HPC) software is still challenging due to the lack of HPC-specific support. In this paper, we design the LM4HPC framework to facilitate the research and development of HPC software analyses and optimizations using LMs. Tailored for supporting HPC datasets, AI models, and pipelines, our framework is built on top of a range of components from different levels of the machine learning software stack, with Hugging Face-compatible APIs. Using three representative tasks, we evaluated the prototype of our framework. The results show that LM4HPC can help users quickly evaluate a set of state-of-the-art models and generate insightful leaderboards.

Related papers

• A Survey of Small Language Models [104.80308007044634]
  Small Language Models (SLMs) have become increasingly important due to their efficiency and performance to perform various language tasks with minimal computational resources. We present a comprehensive survey on SLMs, focusing on their architectures, training techniques, and model compression techniques.
  arXiv  Detail & Related papers  (2024-10-25T23:52:28Z)
• Octopus: On-device language model for function calling of software APIs [9.78611123915888]
  Large Language Models (LLMs) play a crucial role due to their advanced text processing and generation abilities. This study introduces a new strategy aimed at harnessing on-device LLMs in invoking software APIs.
  arXiv  Detail & Related papers  (2024-04-02T01:29:28Z)
• Language-Driven Engineering An Interdisciplinary Software Development Paradigm [51.29189754953934]
  Our illustration includes seven graphical Integrated Modeling Environments (IMEs) that support full code generation. Four browser-based applications that were modeled and then fully automatically generated and produced using DIME. Our technology is open source and the products presented are currently in use.
  arXiv  Detail & Related papers  (2024-02-16T13:37:57Z)
• The Landscape and Challenges of HPC Research and LLMs [12.57518012358534]
  Large language models (LLMs) have revolutionized the field of deep learning. encoder-decoder models and prompt-based techniques have shown immense potential for natural language processing and code-based tasks.
  arXiv  Detail & Related papers  (2024-02-03T04:21:07Z)
• MonoCoder: Domain-Specific Code Language Model for HPC Codes and Tasks [5.125171374181664]
  A growing trend in AI for software development is to develop large language models (LLMs) to address a variety of programming tasks. Even LLMs applied to tasks from the high-performance computing ( HPC) domain are huge in size and demand expensive compute resources for training. This is partly because LLMs for HPC tasks are obtained by finetuning existing LLMs that support several natural and/or programming languages. We build an HPC-specific LM, named MonoCoder, which is orders of magnitude smaller than existing LMs but delivers better performance on non- HPC and HPC codes.
  arXiv  Detail & Related papers  (2023-12-20T15:11:06Z)
• What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning [115.19451843294154]
  Visual instruction tuning is an essential approach to improving the zero-shot generalization capability of Multi-modal Large Language Models (MLLMs) We propose a systematic approach to automatically creating high-quality complex visual reasoning instructions. Our dataset consistently enhances the performance of all the compared MLLMs, e.g., improving the performance of MiniGPT-4 and BLIP-2 on MME-Cognition by 32.6% and 28.8%, respectively.
  arXiv  Detail & Related papers  (2023-11-02T15:36:12Z)
• HPC-GPT: Integrating Large Language Model for High-Performance Computing [3.8078849170829407]
  We propose HPC-GPT, a novel LLaMA-based model that has been supervised fine-tuning using generated QA (Question-Answer) instances for the HPC domain. To evaluate its effectiveness, we concentrate on two HPC tasks: managing AI models and datasets for HPC, and data race detection. Our experiments on open-source benchmarks yield extensive results, underscoring HPC-GPT's potential to bridge the performance gap between LLMs and HPC-specific tasks.
  arXiv  Detail & Related papers  (2023-10-03T01:34:55Z)
• Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data? [49.688233418425995]
  Struc-Bench is a comprehensive benchmark featuring prominent Large Language Models (LLMs) We propose two innovative metrics, P-Score (Prompting Score) and H-Score (Heuristical Score) Our experiments show that applying our structure-aware fine-tuning to LLaMA-7B leads to substantial performance gains.
  arXiv  Detail & Related papers  (2023-09-16T11:31:58Z)
• HPC-Coder: Modeling Parallel Programs using Large Language Models [2.3101915391170573]
  We show how large language models can be applied to tasks specific to high performance and scientific codes. We introduce a new dataset of HPC and scientific codes and use it to fine-tune several pre-trained models. In our experiments, we show that this model can auto-complete HPC functions where generic models cannot.
  arXiv  Detail & Related papers  (2023-06-29T19:44:55Z)
• Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures [67.47328776279204]
  This work introduces a framework to develop efficient, portable Deep Learning and High Performance Computing kernels. We decompose the kernel development in two steps: 1) Expressing the computational core using Processing Primitives (TPPs) and 2) Expressing the logical loops around TPPs in a high-level, declarative fashion. We demonstrate the efficacy of our approach using standalone kernels and end-to-end workloads that outperform state-of-the-art implementations on diverse CPU platforms.
  arXiv  Detail & Related papers  (2023-04-25T05:04:44Z)
• Quantitatively Assessing the Benefits of Model-driven Development in Agent-based Modeling and Simulation [80.49040344355431]
  This paper compares the use of MDD and ABMS platforms in terms of effort and developer mistakes. The obtained results show that MDD4ABMS requires less effort to develop simulations with similar (sometimes better) design quality than NetLogo.
  arXiv  Detail & Related papers  (2020-06-15T23:29:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.