DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

• URL: http://arxiv.org/abs/2607.24717v1
• Date: Mon, 27 Jul 2026 17:54:12 GMT
• Title: DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data
• Abstract summary: We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example.<n>We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks.
• Score: 27.78341478407496
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

Related papers

• Training LLMs for Multi-Step Tool Orchestration with Constrained Data Synthesis and Graduated Rewards [76.49428173793386]
  LLMs must invoke multiple dependent APIs in the correct order while propagating intermediate outputs.<n>Training models to handle such faces two obstacles: existing environments focus on simple per-turn function calls with simulated data, and binary rewards.<n>We present a framework addressing both challenges. First, we construct a reinforcement learning environment backed by a large-scale cache of real API responses, enabling a data synthesis pipeline that samples valid multi-step orchestration traces.<n>Second, we propose a graduated reward design that decomposes correctness into atomic validity and orchestration.
  arXiv  Detail & Related papers  (2026-03-25T18:31:39Z)
• Mathematical Foundations of Modeling ETL Process Chains [10.03291311568095]
  Extract-Transform-Load processes are core components of modern data processing infrastructures.<n>We develop a mathematical model of process chains that is accurate at the level of time-aggregated throughput.<n>We postulate a Flow Balance linking number of threads, mean throughput, and mean processing time.
  arXiv  Detail & Related papers  (2026-02-09T09:23:45Z)
• SynthCoder: A Synthetical Strategy to Tune LLMs for Code Completion [7.668823606571788]
  Code completion is a prominent application of Large Language Models (LLMs) in software engineering.<n>This paper proposes SynthCoder, a model that integrates leading industry practices to achieve state-of-the-art on the Fill-in-the-Middle (FIM) code completion task.
  arXiv  Detail & Related papers  (2025-08-21T12:23:49Z)
• Language Models Improve When Pretraining Data Matches Target Tasks [8.935657480912282]
  BETR is a method that selects pretraining documents based on similarity to benchmark training examples.<n>We compare data selection methods by training over 500 models spanning $1019$ to $1022$ FLOPs and fitting scaling laws to them.<n>We find that BETR achieves a 2.1x compute multiplier over DCLM-Baseline and improves performance on 9 out of 10 tasks across all scales.
  arXiv  Detail & Related papers  (2025-07-16T17:59:45Z)
• Leveraging Machine Learning and Enhanced Parallelism Detection for BPMN Model Generation from Text [75.77648333476776]
  This paper introduces an automated pipeline for extracting BPMN models from text.<n>A key contribution of this work is the introduction of a newly annotated dataset.<n>We augment the dataset with 15 newly annotated documents containing 32 parallel gateways for model training.
  arXiv  Detail & Related papers  (2025-07-11T07:25:55Z)
• Training-Free Dataset Pruning for Instance Segmentation [35.124251909622025]
  Instance segmentation presents three key challenges: pixel-level annotations, instance area variations, and class imbalances.<n>We propose a novel Training-Free dataset Pruning (TFDP) method for instance segmentation.<n>We achieve state-of-the-art results on VOC 2012, Cityscapes, and COCO datasets, generalizing well across CNN and Transformer architectures.
  arXiv  Detail & Related papers  (2025-03-02T10:05:59Z)
• Improving Pretraining Data Using Perplexity Correlations [56.41097718862742]
  We present a framework that selects high-quality pretraining data without any LLM training of our own.<n>We build a new statistical framework for data selection centered around estimates of perplexity-benchmark correlations.<n>Our approach outperforms DSIR on every benchmark, while matching the best data selector found in DataComp-LM.
  arXiv  Detail & Related papers  (2024-09-09T17:23:29Z)
• An Integrated Data Processing Framework for Pretraining Foundation Models [57.47845148721817]
  Researchers and practitioners often have to manually curate datasets from difference sources. We propose a data processing framework that integrates a Processing Module and an Analyzing Module. The proposed framework is easy to use and highly flexible.
  arXiv  Detail & Related papers  (2024-02-26T07:22:51Z)
• How to Train Data-Efficient LLMs [56.41105687693619]
  We study data-efficient approaches for pre-training language models (LLMs) We find that Ask-LLM and Density sampling are the best methods in their respective categories. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories.
  arXiv  Detail & Related papers  (2024-02-15T02:27:57Z)
• SOTASTREAM: A Streaming Approach to Machine Translation Training [13.39347756245191]
  Many machine translation toolkits make use of a data preparation step wherein raw data is transformed into a tensor format that can be used directly by the trainer. We propose an alternative approach that separates the generation of data from the consumption of that data. In this approach, there is no separate pre-processing step; data generation produces an infinite stream of permutations of the raw training data.
  arXiv  Detail & Related papers  (2023-08-14T22:47:19Z)
• Bellamy: Reusing Performance Models for Distributed Dataflow Jobs Across Contexts [52.9168275057997]
  This paper presents Bellamy, a novel modeling approach that combines scale-outs, dataset sizes, and runtimes with additional descriptive properties of a dataflow job. We evaluate our approach on two publicly available datasets consisting of execution data from various dataflow jobs carried out in different environments.
  arXiv  Detail & Related papers  (2021-07-29T11:57:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.