UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

• URL: http://arxiv.org/abs/2607.08646v1
• Date: Thu, 09 Jul 2026 16:18:07 GMT
• Title: UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing
• Abstract summary: UltraX is a function-calling refinement framework for large-scale pre-training data.<n>It completes the editing function space by introducing insertion in addition to deletion and modification.<n>Experiments show that UltraX achieves the highest average performance across all corpora.
• Score: 33.19471537533819
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on data expansion and more on higher-quality data utilization. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert original-refined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments show that UltraX achieves the highest average performance across all corpora and also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.

Related papers

• DataEvolver: Automatic Data Preparation for Large Language Models through Multi-Level Self-Evolving [55.85201665674226]
  High-quality training data is essential to large language models (LLMs)<n>Existing automatic data preparation methods rely on predefined pipelines or customized human instructions.<n>We introduce DataEvolver, the first self-evolving data preparation system that automatically constructs pipelines to transform raw data into high-quality data.
  arXiv  Detail & Related papers  (2026-06-05T07:44:34Z)
• VIP-COP: Context Optimization for Tabular Foundation Models [23.854424850102273]
  VIP-COP estimates the Value of Importance for Prediction of training examples and features for hard Context OPtimization for TFMs.<n>Its explicit selection mechanism suppresses noise and isolates influential data, enabling the model to also benefit from data augmentation.<n> VIP-COP is (i) fast, boosting performance often within minutes of optimization; (ii) budget-aware and any-time, improving with additional test-time compute; (iii) model-aware yet fully black-box, requiring no access to model internals; (iv) interpretable, identifying discrete Very Important Predictors'' (sample
  arXiv  Detail & Related papers  (2026-05-13T02:28:31Z)
• A Systematic Post-Train Framework for Video Generation [76.26555417456773]
  Large-scale video diffusion models have demonstrated impressive capabilities in generating high-resolution and semantically rich content.<n>A significant gap remains between their pretraining performance and real-world deployment requirements due to critical issues such as prompt sensitivity, temporal inconsistency, and prohibitive inference costs.<n>We propose a comprehensive post-training framework that systematically aligns pretrained models with user intentions through four synergistic stages.
  arXiv  Detail & Related papers  (2026-04-28T09:34:51Z)
• Evolving Demonstration Optimization for Chain-of-Thought Feature Transformation [25.08795071237411]
  Feature Transformation is a core data-centric AI task that improves feature space quality to advance predictive performance.<n>Existing solutions rely on discrete search or latent generation, but they are frequently limited by sample inefficiency, invalid candidates, and redundant generations with limited coverage.<n>We propose a framework that optimize context data for LLM-driven FT by evolving trajectory-level experiences in a closed loop.
  arXiv  Detail & Related papers  (2026-02-13T01:12:41Z)
• Fine-tuning Done Right in Model Editing [83.79661791576103]
  Fine-tuning, a foundational method for adapting large language models, has long been considered ineffective for model editing.<n>We restore fine-tuning to the standard breadth-first (i.e., epoch-based) pipeline with mini-batch optimization.<n>We derive LocFT-BF, a simple and effective localized editing method built on the restored fine-tuning framework.
  arXiv  Detail & Related papers  (2025-09-26T08:53:13Z)
• Data Efficient Adaptation in Large Language Models via Continuous Low-Rank Fine-Tuning [34.343514432589586]
  This paper proposes textbf, a novel framework that integrates Low-Rank Adaptation (LoRA) with a continuous fine-tuning strategy.<n> Experiments on 15 diverse datasets show that DEAL consistently outperforms baseline methods.<n>These findings demonstrate the potential of our approach to advance continual adaptation in Large Language Models.
  arXiv  Detail & Related papers  (2025-09-23T12:55:57Z)
• RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs [76.3459242819381]
  RefineX is a novel framework for large-scale, surgical refinement of pre-training data through programmatic editing tasks.<n>The core strength of RefineX lies in distilling high-quality, expert-guided end-to-end refinement results into minimal edit-based deletion programs.<n>We evaluate RefineX across from-scratch pre-training at multiple model scales and find that it consistently outperforms models trained on raw, filtered, or alternatively refined data.
  arXiv  Detail & Related papers  (2025-07-04T02:19:58Z)
• Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models [12.500777267361102]
  We introduce a novel textbfpreference-textbforiented supervised textbffine-textbftuning approach, namely PoFT.<n>The intuition is to boost SFT by imposing a particular preference: textitfavoring the target model over aligned LLMs on the same SFT data.<n>PoFT achieves stable and consistent improvements over the SFT baselines across different training datasets and base models.
  arXiv  Detail & Related papers  (2024-12-17T12:49:14Z)
• Constrain Alignment with Sparse Autoencoders [45.131670081186]
  Feature-level constrained Preference Optimization is a novel method designed to simplify the alignment process while ensuring stability.<n>Our approach enjoys efficiency by using sparse features activated in a well-trained sparse autoencoder and the quality of sequential KL divergence.
  arXiv  Detail & Related papers  (2024-11-12T07:54:13Z)
• Bridging SFT and DPO for Diffusion Model Alignment with Self-Sampling Preference Optimization [67.8738082040299]
  Self-Sampling Preference Optimization (SSPO) is a new alignment method for post-training reinforcement learning.<n>SSPO eliminates the need for paired data and reward models while retaining the training stability of SFT.<n>SSPO surpasses all previous approaches on the text-to-image benchmarks and demonstrates outstanding performance on the text-to-video benchmarks.
  arXiv  Detail & Related papers  (2024-10-07T17:56:53Z)
• In-context Demonstration Matters: On Prompt Optimization for Pseudo-Supervision Refinement [71.60563181678323]
  Large language models (LLMs) have achieved great success across diverse tasks, and fine-tuning is sometimes needed to further enhance generation quality.<n>To handle these challenges, a direct solution is to generate high-confidence'' data from unsupervised downstream tasks.<n>We propose a novel approach, pseudo-supervised demonstrations aligned prompt optimization (PAPO) algorithm, which jointly refines both the prompt and the overall pseudo-supervision.
  arXiv  Detail & Related papers  (2024-10-04T03:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.