Improving Parallel Program Performance Through DSL-Driven Code Generation with LLM Optimizers

• URL: http://arxiv.org/abs/2410.15625v1
• Date: Mon, 21 Oct 2024 04:08:37 GMT
• Title: Improving Parallel Program Performance Through DSL-Driven Code Generation with LLM Optimizers
• Authors: Anjiang Wei, Allen Nie, Thiago S. F. X. Teixeira, Rohan Yadav, Wonchan Lee, Ke Wang, Alex Aiken,
• Abstract summary: Mapping computations to processors and assigning memory are critical for maximizing performance in parallel programming. These mapping decisions are managed through the development of specialized low-level system code, called mappers, crafted by performance engineers. We introduce an approach that leverages recent advances in LLM-baseds for mapper design. In under ten minutes, our method automatically discovers mappers that surpass human expert designs in scientific applications by up to 1.34X speedup.
• Score: 9.880183350366792
• License:
• Abstract: Mapping computations to processors and assigning data to memory are critical for maximizing performance in parallel programming. These mapping decisions are managed through the development of specialized low-level system code, called mappers, crafted by performance engineers. Each mapper is tailored to a specific application and optimized for the underlying machine architecture, a process that requires days of refinement and tuning from an expert. Despite advances in system research, automating mapper generation remains a challenge due to the complexity of making millions of decisions to find the optimal solution and generate the solution as code. We introduce an approach that leverages recent advances in LLM-based optimizers for mapper design. In under ten minutes, our method automatically discovers mappers that surpass human expert designs in scientific applications by up to 1.34X speedup. For parallel matrix multiplication algorithms, our mapper achieves up to 1.31X of the expert-designed solution. To achieve this, we simplify the complexity of low-level code generation by introducing a domain-specific language (DSL) that abstracts the low-level system programming details and defines a structured search space for LLMs to explore. To maximize the application performance, we use an LLM optimizer to improve an agentic system that generates the mapper code. As a result, this approach significantly reduces the workload for performance engineers while achieving substantial performance gains across diverse applications. Finally, our results demonstrate the effectiveness of LLM-based optimization in system design and suggest its potential for addressing other complex system challenges.

Related papers

• LLM-based Optimization of Compound AI Systems: A Survey [64.39860384538338]
  In a compound AI system, components such as an LLM call, a retriever, a code interpreter, or tools are interconnected. Recent advancements enable end-to-end optimization of these parameters using an LLM. This paper presents a survey of the principles and emerging trends in LLM-based optimization of compound AI systems.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• In-the-loop Hyper-Parameter Optimization for LLM-Based Automated Design of Heuristics [0.020482269513546456]
  Large Language Models (LLMs) have shown great potential in automatically generating and optimizing (meta)heuristics. This paper presents a novel hybrid approach, LLaMEA-HPO, which integrates an open source LLaMEA framework with a Hyper- Evolutionary Optimization (HPO) procedure in the loop.
  arXiv  Detail & Related papers  (2024-10-07T14:04:31Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• Archon: An Architecture Search Framework for Inference-Time Techniques [31.655124464284523]
  Archon is a framework for selecting, combining, and stacking layers of inference-time techniques. We evaluate Archon architectures across a range of instruction-following, reasoning, and coding benchmarks.
  arXiv  Detail & Related papers  (2024-09-23T17:53:42Z)
• Rome was Not Built in a Single Step: Hierarchical Prompting for LLM-based Chip Design [22.70660876673987]
  Large Language Models (LLMs) are effective in computer hardware synthesis via hardware description language (HDL) generation. However, LLM-assisted approaches for HDL generation struggle when handling complex tasks. We introduce a suite of hierarchical prompting techniques which facilitate efficient stepwise design methods.
  arXiv  Detail & Related papers  (2024-07-23T21:18:31Z)
• Iterative or Innovative? A Problem-Oriented Perspective for Code Optimization [81.88668100203913]
  Large language models (LLMs) have demonstrated strong capabilities in solving a wide range of programming tasks. In this paper, we explore code optimization with a focus on performance enhancement, specifically aiming to optimize code for minimal execution time.
  arXiv  Detail & Related papers  (2024-06-17T16:10:10Z)
• Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmark [166.40879020706151]
  This paper proposes a shift towards BP-free, zeroth-order (ZO) optimization as a solution for reducing memory costs during fine-tuning. Unlike traditional ZO-SGD methods, our work expands the exploration to a wider array of ZO optimization techniques. Our study unveils previously overlooked optimization principles, highlighting the importance of task alignment, the role of the forward gradient method, and the balance between algorithm complexity and fine-tuning performance.
  arXiv  Detail & Related papers  (2024-02-18T14:08:48Z)
• Can LLMs Configure Software Tools [0.76146285961466]
  In software engineering, the meticulous configuration of software tools is crucial in ensuring optimal performance within intricate systems. In this study, we embark on an exploration of leveraging Large-Language Models (LLMs) to streamline the software configuration process. Our work presents a novel approach that employs LLMs, such as Chat-GPT, to identify starting conditions and narrow down the search space, improving configuration efficiency.
  arXiv  Detail & Related papers  (2023-12-11T05:03:02Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• Multi-Agent Reinforcement Learning for Microprocessor Design Space Exploration [71.95914457415624]
  Microprocessor architects are increasingly resorting to domain-specific customization in the quest for high-performance and energy-efficiency. We propose an alternative formulation that leverages Multi-Agent RL (MARL) to tackle this problem. Our evaluation shows that the MARL formulation consistently outperforms single-agent RL baselines.
  arXiv  Detail & Related papers  (2022-11-29T17:10:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.