Large Language Models Synergize with Automated Machine Learning

• URL: http://arxiv.org/abs/2405.03727v3
• Date: Mon, 9 Sep 2024 15:04:15 GMT
• Title: Large Language Models Synergize with Automated Machine Learning
• Authors: Jinglue Xu, Jialong Li, Zhen Liu, Nagar Anthel Venkatesh Suryanarayanan, Guoyuan Zhou, Jia Guo, Hitoshi Iba, Kenji Tei,
• Abstract summary: This paper explores a novel form of program synthesis, targeting machine learning (ML) programs, by combining large language models (LLMs) and automated machine learning (autoML) In experiments, given the textual task description, our method, Text-to-ML, generates the complete and optimized ML program in a fully autonomous process.
• Score: 12.364087286739647
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, program synthesis driven by large language models (LLMs) has become increasingly popular. However, program synthesis for machine learning (ML) tasks still poses significant challenges. This paper explores a novel form of program synthesis, targeting ML programs, by combining LLMs and automated machine learning (autoML). Specifically, our goal is to fully automate the generation and optimization of the code of the entire ML workflow, from data preparation to modeling and post-processing, utilizing only textual descriptions of the ML tasks. To manage the length and diversity of ML programs, we propose to break each ML program into smaller, manageable parts. Each part is generated separately by the LLM, with careful consideration of their compatibilities. To ensure compatibilities, we design a testing technique for ML programs. Unlike traditional program synthesis, which typically relies on binary evaluations (i.e., correct or incorrect), evaluating ML programs necessitates more than just binary judgments. Our approach automates the numerical evaluation and optimization of these programs, selecting the best candidates through autoML techniques. In experiments across various ML tasks, our method outperforms existing methods in 10 out of 12 tasks for generating ML programs. In addition, autoML significantly improves the performance of the generated ML programs. In experiments, given the textual task description, our method, Text-to-ML, generates the complete and optimized ML program in a fully autonomous process. The implementation of our method is available at https://github.com/JLX0/llm-automl.

Related papers

• Adaptive Self-improvement LLM Agentic System for ML Library Development [8.766639641127412]
  Large language models (LLMs) have shown general coding capabilities. LLMs need complex reasoning with limited data in order to complete this task. We introduce an adaptive self-improvement agentic system to generate ASPL code with both open and closed-source LLMs.
  arXiv  Detail & Related papers  (2025-02-04T17:57:17Z)
• Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey [3.340984908213717]
  Building effective machine learning (ML) to address complex tasks is a primary focus of the Automatic ML (AutoML) community. Recently, the integration of Large Language Models (LLMs) into ML has shown great potential for automating and enhancing various stages of the ML pipeline.
  arXiv  Detail & Related papers  (2024-11-11T21:54:26Z)
• DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Execution [114.61347672265076]
  Development of MLLMs for real-world robots is challenging due to the typically limited computation and memory capacities available on robotic platforms. We propose a Dynamic Early-Exit Framework for Robotic Vision-Language-Action Model (DeeR) that automatically adjusts the size of the activated MLLM. DeeR demonstrates significant reductions in computational costs of LLM by 5.2-6.5x and GPU memory of LLM by 2-6x without compromising performance.
  arXiv  Detail & Related papers  (2024-11-04T18:26:08Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• A Large-Scale Study of Model Integration in ML-Enabled Software Systems [4.776073133338119]
  Machine learning (ML) and its embedding in systems has drastically changed the engineering of software-intensive systems. Traditionally, software engineering focuses on manually created artifacts such as source code and the process of creating them. We present the first large-scale study of real ML-enabled software systems, covering over 2,928 open source systems on GitHub.
  arXiv  Detail & Related papers  (2024-08-12T15:28:40Z)
• Verbalized Machine Learning: Revisiting Machine Learning with Language Models [63.10391314749408]
  We introduce the framework of verbalized machine learning (VML) VML constrains the parameter space to be human-interpretable natural language. We empirically verify the effectiveness of VML, and hope that VML can serve as a stepping stone to stronger interpretability.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)
• Position: A Call to Action for a Human-Centered AutoML Paradigm [83.78883610871867]
  Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) We argue that a key to unlocking AutoML's full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems.
  arXiv  Detail & Related papers  (2024-06-05T15:05:24Z)
• Tool Learning in the Wild: Empowering Language Models as Automatic Tool Agents [56.822238860147024]
  Augmenting large language models with external tools has emerged as a promising approach to extend their utility. Previous methods manually parse tool documentation and create in-context demonstrations, transforming tools into structured formats for LLMs to use in their step-by-step reasoning. We propose AutoTools, a framework that enables LLMs to automate the tool-use workflow.
  arXiv  Detail & Related papers  (2024-05-26T11:40:58Z)
• AutoMMLab: Automatically Generating Deployable Models from Language Instructions for Computer Vision Tasks [39.71649832548044]
  AutoMMLab is a general-purpose LLM-empowered AutoML system that follows user's language instructions. The proposed AutoMMLab system effectively employs LLMs as the bridge to connect AutoML and OpenMMLab community. Experiments show that our AutoMMLab system is versatile and covers a wide range of mainstream tasks.
  arXiv  Detail & Related papers  (2024-02-23T14:38:19Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks. This paper presents the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)
• Machine Learning for Software Engineering: A Tertiary Study [13.832268599253412]
  Machine learning (ML) techniques increase the effectiveness of software engineering (SE) lifecycle activities. We systematically collected, quality-assessed, summarized, and categorized 83 reviews in ML for SE published between 2009-2022, covering 6,117 primary studies. The SE areas most tackled with ML are software quality and testing, while human-centered areas appear more challenging for ML.
  arXiv  Detail & Related papers  (2022-11-17T09:19:53Z)
• Operationalizing Machine Learning: An Interview Study [13.300075655862573]
  We conduct semi-structured interviews with 18 machine learning engineers (MLEs) working across many applications. Our interviews expose three variables that govern success for a production ML deployment: Velocity, Validation, and Versioning. We summarize common practices for successful ML experimentation, deployment, and sustaining production performance.
  arXiv  Detail & Related papers  (2022-09-16T16:59:36Z)
• MLGO: a Machine Learning Guided Compiler Optimizations Framework [0.0]
  This work is the first full integration of machine learning in a complex compiler pass in a real-world setting. We use two different ML algorithms to train the inlining-for-size model, and achieve up to 7% size reduction. The same model generalizes well to a diversity of real-world targets, as well as to the same set of targets after months of active development.
  arXiv  Detail & Related papers  (2021-01-13T00:02:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.