BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training

• URL: http://arxiv.org/abs/2605.09994v2
• Date: Fri, 15 May 2026 02:55:12 GMT
• Title: BatchWeave: A Consistent Object-Store-Native Data Plane for Large Foundation Model Training
• Abstract summary: BatchWeave is an object-store-native training data plane for distributed LFM training.<n>It implements versioned manifests and conditional object writes to coordinate batch publication, recovery, and lifecycle management.<n>It provides full failure isolation, outperforms Apache Kafka in ingestion throughput, and achieves lower consumer read latency than Kafka.
• Score: 13.09314762856332
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern Large Foundation Model (LFM) training has transformed the data pipeline from a static ingestion layer into a dynamic component that must co-evolve with the training process. Existing systems are ill-equipped: colocated dataloaders offer no failure isolation, while message queue-based disaggregated dataloaders operate on a record/offset abstraction that cannot express the batch-level semantics required by distributed training. We present BatchWeave, an object-store-native training data plane for distributed LFM training. BatchWeave uses versioned manifests and conditional object writes to coordinate batch publication, recovery, and lifecycle management. First, it introduces the Transactional Global Batch (TGB), which builds on versioned-manifest ACID storage semantics and extends them with training-specific consistency, including atomic all-rank batch visibility, a globally ordered step sequence, checkpoint-aligned lifecycle management, and end-to-end exactly-once recovery. Second, it realizes recovery and retention directly in the storage layer, by durably persisting producer state through the commit protocol and tying reclamation to distributed checkpoint state. Third, its Decentralized Adaptive Commit (DAC) algorithm sustains stable ingestion throughput as the manifest grows, without any inter-producer communication. Evaluations on large-scale multimodal pre-training and SFT workloads using 64 GPUs show that BatchWeave outperforms colocated dataloader throughput while providing full failure isolation, outperforms Apache Kafka in ingestion throughput, and achieves lower consumer read latency than Kafka.

Related papers

• FedCVESA: Taking Away Training Data in Federated Learning via Correlation Value Encoding and Segmented Aggregation [13.470122153087011]
  Malicious training could abuse the memorization capacity of deep models to store and later recover training data.<n>We study a whitebox TATD attack in which a malicious server selects n target clients from participating clients and actively writes private training data into the global model.<n>Experiments show that the proposed method can steal semantically meaningful private training images from the trained model.
  arXiv  Detail & Related papers  (2026-07-08T11:55:00Z)
• FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models [31.501910341506164]
  FlowTrain is a flow-based decoupled training framework for vision-language computation models.<n>It reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool.<n>Experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7x throughput improvement.
  arXiv  Detail & Related papers  (2026-06-22T09:33:44Z)
• Versioned Late Materialization for Ultra-Long Sequence Training in Recommendation Systems at Scale [12.736059967085025]
  Modern Deep Learning Recommendation Models (DLRMs) follow scaling laws with sequence length.<n>"Fat Row" paradigm pre-materializes these sequences into every training example, creating a storage and I/O wall.<n>We present a emphversioned late materialization paradigm that eliminates this redundancy by storing UIH once in a normalized, immutable tier.
  arXiv  Detail & Related papers  (2026-04-27T06:41:39Z)
• SENTINEL: Stagewise Integrity Verification for Pipeline Parallel Decentralized Training [54.8494905524997]
  Decentralized training introduces critical security risks when executed across untrusted, geographically distributed nodes.<n>We propose SENTINEL, a verification mechanism for pipeline parallelism (PP) training without duplication.<n>Experiments demonstrate successful training of up to 4B- parameter LLMs across untrusted distributed environments with up to 176 workers while maintaining model convergence and performance.
  arXiv  Detail & Related papers  (2026-03-03T23:51:10Z)
• Pull Requests as a Training Signal for Repo-Level Code Editing [49.82435173554125]
  Clean Pull Request (Clean-PR) is a mid-training paradigm that leverages real-world GitHub pull requests as a training signal for repository-level editing.<n>We introduce a scalable pipeline that converts noisy pull request diffs into Search/Replace edit blocks through reconstruction and validation.<n>On SWE-bench, our model significantly outperforms the instruction-tuned baseline, achieving absolute improvements of 13.6% on SWE-bench Lite and 12.3% on SWE-bench Verified.
  arXiv  Detail & Related papers  (2026-02-07T09:22:25Z)
• Nimbus: A Unified Embodied Synthetic Data Generation Framework [51.55989844555466]
  Scaling data volume and diversity is critical for generalizing embodied intelligence.<n>We present Nimbus, a unified synthetic data generation framework designed to integrate heterogeneous navigation and manipulation pipelines.<n>Our evaluation demonstrates that Nimbus achieves a 2-3X improvement in end-to-end throughput compared to unoptimized baselines.
  arXiv  Detail & Related papers  (2026-01-29T09:27:31Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• The Streaming Batch Model for Efficient and Fault-Tolerant Heterogeneous Execution [28.768566833298365]
  We present Ray Data, a streaming batch system that improves throughput on heterogeneous batch inference pipelines by 2.5-12$times$.<n>Ray Data improves training throughput for multimodal models such as Stable Diffusion by 31% compared to single-node ML data loaders.
  arXiv  Detail & Related papers  (2025-01-16T19:54:01Z)
• CaBaFL: Asynchronous Federated Learning via Hierarchical Cache and Feature Balance [23.125185494897522]
  Federated Learning (FL) as a promising distributed machine learning paradigm has been widely adopted in Artificial Intelligence of Things (AIoT) applications. The efficiency and inference capability of FL is seriously limited due to the presence of stragglers and data imbalance across massive AIoT devices. We present a novel FL approach named CaBaFL, which includes a hierarchical Cache-based aggregation mechanism and a feature Balance-guided device selection strategy.
  arXiv  Detail & Related papers  (2024-04-19T12:39:11Z)
• Subspace based Federated Unlearning [75.90552823500633]
  Federated unlearning (FL) aims to remove a specified target client's contribution in FL to satisfy the user's right to be forgotten. Most existing federated unlearning algorithms require the server to store the history of the parameter updates. We propose a simple-yet-effective subspace based federated unlearning method, dubbed SFU, that lets the global model perform gradient ascent.
  arXiv  Detail & Related papers  (2023-02-24T04:29:44Z)
• Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning [60.41501515192088]
  Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively. The data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model. In this work, we integrate the local real data with the global gradient prototypes to form the local balanced datasets.
  arXiv  Detail & Related papers  (2023-01-25T03:18:10Z)
• TensAIR: Real-Time Training of Neural Networks from Data-streams [1.409180142531996]
  This paper presents TensAIR, the first OL system for training ANNs in real time. TensAIR achieves remarkable performance and scalability by using a decentralized and asynchronous architecture to train ANN models. We empirically demonstrate that TensAIR achieves a nearly linear scale-out performance in terms of (1) the number of worker nodes deployed in the network, and (2) the throughput at which the data batches arrive.
  arXiv  Detail & Related papers  (2022-11-18T15:11:44Z)
• RecD: Deduplication for End-to-End Deep Learning Recommendation Model Training Infrastructure [3.991664287163157]
  RecD addresses immense storage, preprocessing, and training overheads caused by feature duplication inherent in industry-scale DLRM training datasets. We show how RecD improves the training and preprocessing throughput and storage efficiency by up to 2.48x, 1.79x, and 3.71x, respectively, in an industry-scale DLRM training system.
  arXiv  Detail & Related papers  (2022-11-09T22:21:19Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• How Well Self-Supervised Pre-Training Performs with Streaming Data? [73.5362286533602]
  In real-world scenarios where data are collected in a streaming fashion, the joint training scheme is usually storage-heavy and time-consuming. It is unclear how well sequential self-supervised pre-training performs with streaming data. We find sequential self-supervised learning exhibits almost the same performance as the joint training when the distribution shifts within streaming data are mild.
  arXiv  Detail & Related papers  (2021-04-25T06:56:48Z)
• Training Recommender Systems at Scale: Communication-Efficient Model and Data Parallelism [56.78673028601739]
  We propose a compression framework called Dynamic Communication Thresholding (DCT) for communication-efficient hybrid training. DCT reduces communication by at least $100times$ and $20times$ during DP and MP, respectively. It improves end-to-end training time for a state-of-the-art industrial recommender model by 37%, without any loss in performance.
  arXiv  Detail & Related papers  (2020-10-18T01:44:42Z)
• Unshuffling Data for Improved Generalization [65.57124325257409]
  Generalization beyond the training distribution is a core challenge in machine learning. We show that partitioning the data into well-chosen, non-i.i.d. subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization.
  arXiv  Detail & Related papers  (2020-02-27T03:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.