Towards Next-Generation LLM Training: From the Data-Centric Perspective

• URL: http://arxiv.org/abs/2603.14712v1
• Date: Mon, 16 Mar 2026 01:40:09 GMT
• Title: Towards Next-Generation LLM Training: From the Data-Centric Perspective
• Abstract summary: Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances.<n>Despite this success, the preparation and effective utilization of the massive datasets required for LLM training remain major bottlenecks.<n>We propose building a robust, agent-based automatic data preparation system that supports automated workflow construction and scalable data management.<n>We argue for a unified data-model interaction training system in which data is dynamically selected, mixed, and reweighted throughout the training process, enabling more efficient, adaptive, and performance-aware data utilization.
• Score: 29.115711465172122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances. Despite this success, the preparation and effective utilization of the massive datasets required for LLM training remain major bottlenecks. In current practice, LLM training data is often constructed using ad hoc scripts, and there is still a lack of mature, agent-based data preparation systems that can automatically construct robust and reusable data workflows, thereby freeing data scientists from repetitive and error-prone engineering efforts. Moreover, once collected, datasets are often consumed largely in their entirety during training, without systematic mechanisms for data selection, mixture optimization, or reweighting. To address these limitations, we advocate two complementary research directions. First, we propose building a robust, agent-based automatic data preparation system that supports automated workflow construction and scalable data management. Second, we argue for a unified data-model interaction training system in which data is dynamically selected, mixed, and reweighted throughout the training process, enabling more efficient, adaptive, and performance-aware data utilization. Finally, we discuss the remaining challenges and outline promising directions for future research and system development.

Related papers

• Demystifying Data Organization for Enhanced LLM Training [27.639372382009725]
  Large Language Models (LLMs) have revolutionized various fields, yet their training efficiency is heavily reliant on effective data curation.<n>This paper explores the influence of data organization on LLM training by reusing pre-computed sample-level scores originally generated for data efficiency.
  arXiv  Detail & Related papers  (2026-05-28T17:58:53Z)
• Data Science and Technology Towards AGI Part I: Tiered Data Management [53.64581824953229]
  We argue that the development of artificial intelligence is entering a new phase of data-model co-evolution.<n>We introduce an L0-L4 tiered data management framework, ranging from raw uncurated resources to organized and verifiable knowledge.<n>We validate the effectiveness of the proposed framework through empirical studies.
  arXiv  Detail & Related papers  (2026-02-09T18:47:51Z)
• Can LLMs Clean Up Your Mess? A Survey of Application-Ready Data Preparation with LLMs [66.63911043019294]
  Data preparation aims to denoise raw datasets, uncover cross-dataset relationships, and extract valuable insights from them.<n>This paper focuses on the use of LLM techniques to prepare data for diverse downstream tasks.<n>We introduce a task-centric taxonomy that organizes the field into three major tasks: data cleaning, standardization, error processing, imputation, data integration, and data enrichment.
  arXiv  Detail & Related papers  (2026-01-22T12:02:45Z)
• History Is Not Enough: An Adaptive Dataflow System for Financial Time-Series Synthesis [25.486090554711797]
  "History Is Not Enough" underscores the need for adaptive data generation that learns to evolve with the market.<n>We present a drift-aware dataflow system that integrates machine learning-based adaptive control into the data curation process.
  arXiv  Detail & Related papers  (2026-01-15T07:38:59Z)
• A Survey on Efficient Large Language Model Training: From Data-centric Perspectives [42.897899343082806]
  We present the first systematic survey of data-efficient Large Language Models post-training from a data-centric perspective.<n>We propose a taxonomy of data-efficient LLM post-training methods, covering data selection, data quality enhancement, synthetic data generation, data distillation and compression, and self-evolving data ecosystems.<n>We hope our work inspires further exploration into maximizing the potential of data utilization in large-scale model training.
  arXiv  Detail & Related papers  (2025-10-29T17:01:55Z)
• Towards an Introspective Dynamic Model of Globally Distributed Computing Infrastructures [27.473508984130728]
  Large-scale scientific collaborations generate petabytes of data, with volumes soon expected to reach exabytes.<n>To manage these computational and storage demands, centralized workflow and data management systems are implemented.<n>A significant obstacle in adopting more effective or AI-driven solutions is the absence of a quick and reliable introspective dynamic model.
  arXiv  Detail & Related papers  (2025-06-24T12:42:36Z)
• LAM SIMULATOR: Advancing Data Generation for Large Action Model Training via Online Exploration and Trajectory Feedback [121.78866929908871]
  Large Action Models (LAMs) for AI Agents offer incredible potential but face challenges due to the need for high-quality training data.<n>We present LAM SIMULATOR, a comprehensive framework designed for online exploration of agentic tasks with high-quality feedback.<n>Our framework features a dynamic task query generator, an extensive collection of tools, and an interactive environment where Large Language Model (LLM) Agents can call tools and receive real-time feedback.
  arXiv  Detail & Related papers  (2025-06-02T22:36:02Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• Data Management For Training Large Language Models: A Survey [64.18200694790787]
  Data plays a fundamental role in training Large Language Models (LLMs) This survey aims to provide a comprehensive overview of current research in data management within both the pretraining and supervised fine-tuning stages of LLMs.
  arXiv  Detail & Related papers  (2023-12-04T07:42:16Z)
• Fix your Models by Fixing your Datasets [0.6058427379240697]
  Current machine learning tools lack streamlined processes for improving the data quality. We introduce a systematic framework for finding noisy or mislabelled samples in the dataset. We demonstrate the efficacy of our framework on public as well as private enterprise datasets of two Fortune 500 companies.
  arXiv  Detail & Related papers  (2021-12-15T02:41:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.