ForgeTrain: Forging Production-Grade Training Frameworks via Harness-Driven AI Development

• URL: http://arxiv.org/abs/2609.13645v1
• Date: Sat, 12 Sep 2026 01:47:57 GMT
• Title: ForgeTrain: Forging Production-Grade Training Frameworks via Harness-Driven AI Development
• Abstract summary: Training large models still relies on general-purpose frameworks such as Megatron-LM.<n>We propose Forge Engineering: building a dedicated implementation from scratch for each scenario.<n>We instantiate this paradigm for training frameworks as ForgeTrain, which holds a trusted framework as a golden reference.
• Score: 31.93593134525649
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large models still relies on general-purpose frameworks such as Megatron-LM, whose generality tax constrains scenario-specific optimization and adds runtime overhead through accumulated abstraction. AI code generation reduces the cost of building a framework, and makes it affordable to forge one per scenario. We propose Forge Engineering: building a dedicated implementation from scratch for each scenario and iteratively optimizing it toward peak performance under correctness and usability constraints. Dedicated implementations inherit no abstraction boundaries, so they can integrate optimizations across the stack and reach a higher performance ceiling. We instantiate this paradigm for training frameworks as ForgeTrain, which holds a trusted framework as a golden reference and relaxes equivalence monotonically from Bit-for-Bit to Surpass. Experiments across multiple model--hardware configurations show that ForgeTrain consistently produces correct training engines and improves MFU over established training frameworks by 4.7--33.2%. To our knowledge this is the first production-grade training framework forged end-to-end by AI to match or surpass its human reference.

Related papers

• ClawGym II: Exploring Black-Box RL on Agent Harness [82.92963070856416]
  Agent harnesses have substantially improved performance on long-horizon tasks by coordinating agent interactions with the environment.<n>We present a unified black-box RL framework for stable and scalable optimization of general agents through complex harnesses.
  arXiv  Detail & Related papers  (2026-08-17T16:53:03Z)
• Self-Evolving Embodied Agents via Skill-Harness Evolution [53.307052568223945]
  Embodied agents are increasingly built as systems around foundation models.<n>While supervised fine-tuning and reinforcement learning can adapt agents to new environments, they require additional data, rewards, and training runs.<n>We propose SHAPER, a self-evolving framework for train-free embodied adaptation that keeps model parameters frozen.
  arXiv  Detail & Related papers  (2026-08-11T18:55:58Z)
• PithTrain: A Compact and Agent-Native MoE Training System [27.857461550954337]
  We build PithTrain, a compact, agent-native MoE training framework.<n>On ATE-Bench, PithTrain enables higher agent-task efficiency, with up to 62% fewer Agent Turns and 64% less Active GPU Time.
  arXiv  Detail & Related papers  (2026-05-29T15:52:58Z)
• Rollback-Free Stable Brick Structures Generation [51.856925926010184]
  Existing approaches rely on external physical simulators during inference to perform rejection sampling and brick-by-brick rollbacks.<n>We propose a reinforcement learning paradigm that shifts physical validity enforcement from test-time correction to training-time policy optimization.<n> Experimental results demonstrate that our approach achieves state-of-the-art generation quality while accelerating inference speed by orders of magnitude.
  arXiv  Detail & Related papers  (2026-05-07T21:06:44Z)
• Grouter: Decoupling Routing from Representation for Accelerated MoE Training [10.767613437794537]
  Grouter is a preemptive routing method that by distilling high-quality structures from fully-trained MoE models and serving as a fixed router for target models.<n> Experiments demonstrate that Grouter achieves superior performance and efficiency which boosts pre-training data utilization by 4.28x and achieves up to 33.5% throughput acceleration.
  arXiv  Detail & Related papers  (2026-02-22T06:09:57Z)
• DiRL: An Efficient Post-Training Framework for Diffusion Language Models [54.405206032785706]
  Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models.<n>Existing methods suffer from computational inefficiency and objective mismatches between training and inference.<n>We introduce DiRL, an efficient post-training framework that tightly integrates FlexAttention-accelerated blockwise training with LMDeploy-optimized inference.
  arXiv  Detail & Related papers  (2025-12-23T08:33:19Z)
• A Multi-Level Framework for Accelerating Training Transformer Models [5.268960238774481]
  Training large-scale deep learning models poses an unprecedented demand for computing power. We propose a multi-level framework for training acceleration based on Coalescing, De-coalescing and Interpolation. We prove that the proposed framework reduces the computational cost by about 20% on training BERT/GPT-Base models and up to 51.6% on training the BERT-Large model.
  arXiv  Detail & Related papers  (2024-04-07T03:04:34Z)
• Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training [17.556432199389615]
  Slapo is a schedule language that decouples the execution of a tensor-level operator from its arithmetic definition. We show that Slapo can improve training throughput by up to 2.92x on a single machine with 8 NVIDIA V100 GPUs.
  arXiv  Detail & Related papers  (2023-02-16T00:34:53Z)
• Adversarial Self-Attention for Language Understanding [89.265747130584]
  This paper proposes textitAdversarial Self-Attention mechanism (ASA). ASA adversarially reconstructs the Transformer attentions and facilitates model training from contaminated model structures. For fine-tuning, ASA-empowered models consistently outweigh naive models by a large margin considering both generalization and robustness.
  arXiv  Detail & Related papers  (2022-06-25T09:18:10Z)
• DSEE: Dually Sparsity-embedded Efficient Tuning of Pre-trained Language Models [152.29364079385635]
  As pre-trained models grow bigger, the fine-tuning process can be time-consuming and computationally expensive. We propose a framework for resource- and parameter-efficient fine-tuning by leveraging the sparsity prior in both weight updates and the final model weights. Our proposed framework, dubbed Dually Sparsity-Embedded Efficient Tuning (DSEE), aims to achieve two key objectives: (i) parameter efficient fine-tuning and (ii) resource-efficient inference.
  arXiv  Detail & Related papers  (2021-10-30T03:29:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.