MLZero: A Multi-Agent System for End-to-end Machine Learning Automation

• URL: http://arxiv.org/abs/2505.13941v1
• Date: Tue, 20 May 2025 05:20:53 GMT
• Title: MLZero: A Multi-Agent System for End-to-end Machine Learning Automation
• Authors: Haoyang Fang, Boran Han, Nick Erickson, Xiyuan Zhang, Su Zhou, Anirudh Dagar, Jiani Zhang, Ali Caner Turkmen, Cuixiong Hu, Huzefa Rangwala, Ying Nian Wu, Bernie Wang, George Karypis,
• Abstract summary: We introduce MLZero, a novel multi-agent framework powered by Large Language Models (LLMs)<n>A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context.<n> MLZero demonstrates superior performance on MLE-Bench Lite, outperforming all competitors in both success rate and solution quality.
• Score: 48.716299953336346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing AutoML systems have advanced the automation of machine learning (ML); however, they still require substantial manual configuration and expert input, particularly when handling multimodal data. We introduce MLZero, a novel multi-agent framework powered by Large Language Models (LLMs) that enables end-to-end ML automation across diverse data modalities with minimal human intervention. A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context that effectively guides the subsequent workflow. To address key limitations of LLMs, such as hallucinated code generation and outdated API knowledge, we enhance the iterative code generation process with semantic and episodic memory. MLZero demonstrates superior performance on MLE-Bench Lite, outperforming all competitors in both success rate and solution quality, securing six gold medals. Additionally, when evaluated on our Multimodal AutoML Agent Benchmark, which includes 25 more challenging tasks spanning diverse data modalities, MLZero outperforms the competing methods by a large margin with a success rate of 0.92 (+263.6\%) and an average rank of 2.28. Our approach maintains its robust effectiveness even with a compact 8B LLM, outperforming full-size systems from existing solutions.

Related papers

• Filter-And-Refine: A MLLM Based Cascade System for Industrial-Scale Video Content Moderation [21.18948097612397]
  This paper introduces an efficient method to transform a generative MLLM into a multimodal classifier using minimal discriminative training data.<n>We then propose a router-ranking cascade system that integrates MLLMs with a lightweight router model.<n>Online evaluations show that our system increases automatic content moderation volume by 41%, while the cascading deployment reduces computational cost to only 1.5% of direct full-scale deployment.
  arXiv  Detail & Related papers  (2025-07-23T04:52:58Z)
• LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generation [12.098817831819078]
  Large Language Models (LLMs) have shown promise in single-robot settings, but their application in multi-robot systems remains largely unexplored.<n>This paper introduces a novel LLM-Aided MARL (LAMARL) approach, which integrates MARL with LLMs, significantly enhancing sample efficiency without requiring manual design.
  arXiv  Detail & Related papers  (2025-06-02T10:59:54Z)
• Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization [65.64108848398696]
  We introduce a preference optimization (PO) process to enhance the multimodal reasoning capabilities of MLLMs.<n>Specifically, we design an automated preference data construction pipeline to create MMPR, a high-quality, large-scale multimodal reasoning preference dataset.<n>We explore integrating PO with MLLMs, developing a simple yet effective method, termed Mixed Preference Optimization (MPO), which boosts multimodal CoT performance.
  arXiv  Detail & Related papers  (2024-11-15T18:59:27Z)
• Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench [17.73279547506514]
  We introduce Multimodal Large Language Model Unlearning Benchmark (MLLMU-Bench), a novel benchmark aimed at advancing the understanding of multimodal machine unlearning.<n>MLLMU-Bench consists of 500 fictitious profiles and 153 profiles for public celebrities, each profile feature over 14 customized question-answer pairs, evaluated from both multimodal (image+text) and unimodal (text) perspectives.<n>Surprisingly, our experiments show that unimodal unlearning algorithms excel in generation and cloze tasks, while multimodal unlearning approaches perform better in classification tasks with multimodal inputs.
  arXiv  Detail & Related papers  (2024-10-29T15:07:23Z)
• LLMs Can Evolve Continually on Modality for X-Modal Reasoning [62.2874638875554]
  Existing methods rely heavily on modal-specific pretraining and joint-modal tuning, leading to significant computational burdens when expanding to new modalities. We propose PathWeave, a flexible and scalable framework with modal-Path sWitching and ExpAnsion abilities. PathWeave performs comparably to state-of-the-art MLLMs while concurrently reducing parameter training burdens by 98.73%.
  arXiv  Detail & Related papers  (2024-10-26T13:19:57Z)
• NVLM: Open Frontier-Class Multimodal LLMs [64.00053046838225]
  We introduce NVLM 1.0, a family of frontier-class multimodal large language models (LLMs) that achieve state-of-the-art results on vision-language tasks. We propose a novel architecture that enhances both training efficiency and multimodal reasoning capabilities. We develop production-grade multimodality for the NVLM-1.0 models, enabling them to excel in vision-language tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:59:06Z)
• Position: A Call to Action for a Human-Centered AutoML Paradigm [83.78883610871867]
  Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) We argue that a key to unlocking AutoML's full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems.
  arXiv  Detail & Related papers  (2024-06-05T15:05:24Z)
• Large Language Models Synergize with Automated Machine Learning [12.364087286739647]
  This paper explores a novel form of program synthesis, targeting machine learning (ML) programs, by combining large language models (LLMs) and automated machine learning (autoML) In experiments, given the textual task description, our method, Text-to-ML, generates the complete and optimized ML program in a fully autonomous process.
  arXiv  Detail & Related papers  (2024-05-06T08:09:46Z)
• LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark [81.42376626294812]
  We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs. We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
  arXiv  Detail & Related papers  (2023-06-11T14:01:17Z)
• Multi-Agent Automated Machine Learning [54.14038920246645]
  We propose multi-agent automated machine learning (MA2ML) to handle joint optimization of modules in automated machine learning (AutoML) MA2ML explicitly assigns credit to each agent according to its marginal contribution to enhance cooperation among modules, and incorporates off-policy learning to improve search efficiency. Experiments show that MA2ML yields the state-of-the-art top-1 accuracy on ImageNet under constraints of computational cost.
  arXiv  Detail & Related papers  (2022-10-17T13:32:59Z)
• Robusta: Robust AutoML for Feature Selection via Reinforcement Learning [24.24652530951966]
  We propose the first robust AutoML framework, Robusta--based on reinforcement learning (RL) We show that the framework is able to improve the model robustness by up to 22% while maintaining competitive accuracy on benign samples.
  arXiv  Detail & Related papers  (2021-01-15T03:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.