Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

• URL: http://arxiv.org/abs/2410.14251v2
• Date: Thu, 20 Feb 2025 03:26:26 GMT
• Title: Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation
• Authors: Shuo Tang, Xianghe Pang, Zexi Liu, Bohan Tang, Rui Ye, Tian Jin, Xiaowen Dong, Yanfeng Wang, Siheng Chen,
• Abstract summary: MATRIX is a multi-agent simulator that automatically generates diverse text-based scenarios. We introduce MATRIX-Gen for controllable and highly realistic data synthesis. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model.
• Score: 51.20656279478878
• License:
• Abstract: Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs.

Related papers

• GME: Improving Universal Multimodal Retrieval by Multimodal LLMs [43.457928045291915]
  Universal Multimodal Retrieval (UMR) aims to enable search across various modalities using a unified model. Previous work has attempted to adopt multimodal large language models (MLLMs) to realize UMR using only text data.
  arXiv  Detail & Related papers  (2024-12-22T04:40:24Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMs [38.281562732050084]
  GenSim2 is a scalable framework for complex and realistic simulation task creation. The pipeline can generate data for up to 100 articulated tasks with 200 objects and reduce the required human efforts. We show a promising usage of GenSim2 that the generated data can be used for zero-shot transfer or co-train with real-world collected data.
  arXiv  Detail & Related papers  (2024-10-04T17:51:33Z)
• Align$^2$LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation [56.75665429851673]
  This paper introduces a novel instruction curation algorithm, derived from two unique perspectives, human and LLM preference alignment. Experiments demonstrate that we can maintain or even improve model performance by compressing synthetic multimodal instructions by up to 90%.
  arXiv  Detail & Related papers  (2024-09-27T08:20:59Z)
• Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator [63.762209407570715]
  Genixer is a comprehensive data generation pipeline consisting of four key steps. A synthetic VQA-like dataset trained with LLaVA1.5 enhances performance on 10 out of 12 multimodal benchmarks. MLLMs trained with task-specific datasets can surpass GPT-4V in generating complex instruction tuning data.
  arXiv  Detail & Related papers  (2023-12-11T09:44:41Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• GenSim: Generating Robotic Simulation Tasks via Large Language Models [34.79613485106202]
  GenSim aims to automatically generate rich simulation environments and expert demonstrations. We use GPT4 to expand the existing benchmark by ten times to over 100 tasks. With minimal sim-to-real adaptation, multitask policies pretrained on GPT4-generated simulation tasks exhibit stronger transfer to unseen long-horizon tasks in the real world.
  arXiv  Detail & Related papers  (2023-10-02T17:23:48Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.