FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models

• URL: http://arxiv.org/abs/2606.23087v1
• Date: Mon, 22 Jun 2026 09:33:44 GMT
• Title: FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models
• Abstract summary: FlowTrain is a flow-based decoupled training framework for vision-language computation models.<n>It reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool.<n>Experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7x throughput improvement.
• Score: 31.501910341506164
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Industrial-grade distributed training of vision-language models (VLMs) remains far less efficient than that of unimodal LLMs. Existing solutions either follow a monolithic design that assigns uniform parallelism to heterogeneous modules or adopt a disaggregated deployment that separates modules while executing them as a batch-synchronized pipeline. In this paper, we highlight that the above solutions are still not sufficient, and VLM training can be further decoupled. To this end, we present FlowTrain, a flow-based decoupled training framework that reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool. The encoder and backbone can progress independently over a global virtual address space. Since this execution decoupling fundamentally changes the optimization objective of allocation and scheduling, FlowTrain further introduces a heterogeneous parallel allocator that assigns module-specific parallelism strategies by solving a throughput matching problem. The dynamic packing scheduler is used to construct balanced microbatches at runtime according to the actual LLM-side computation cost. Extensive experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7x throughput improvement, narrowing the efficiency gap to LLM-only training.

Related papers

• FLARE: Diffusion for Hybrid Language Model [72.60770374799634]
  FLARE is a systematic conversion framework for hybrid-attention large language models.<n>It combines a token-equal AR-and-diffusion objective, hardware-aware kernels, and unified inference, enabling one checkpoint to support both AR-style verified decoding and diffusion-style parallel denoising.<n>Our results suggest practical dLLMs are limited not only by decoding algorithms, but also by transfer data quality and the training inefficiency of current block-diffusion objectives.
  arXiv  Detail & Related papers  (2026-06-01T06:58:15Z)
• Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning [21.32826098306222]
  Existing training frameworks fail to address the unique system-level challenges of multi-agent reinforcement learning.<n>We propose FlexMARL, the first end-to-end training framework that holistically optimize rollout, training, and their orchestration.<n>We show that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.
  arXiv  Detail & Related papers  (2026-02-10T09:27:03Z)
• TIDE: Temporal Incremental Draft Engine for Self-Improving LLM Inference [1.0091292967761423]
  TIDE is a serving-engine-native framework that integrates online draft adaptation directly into high-performance LLM inference systems.<n>TIDE reuses target model hidden states generated during inference as training signals, enabling zero-overhead draft adaptation without reloading the target model.<n>Across diverse real-world workloads, TIDE achieves up to 1.15x throughput improvement over static speculative decoding.
  arXiv  Detail & Related papers  (2026-02-05T00:06:12Z)
• DiRL: An Efficient Post-Training Framework for Diffusion Language Models [54.405206032785706]
  Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models.<n>Existing methods suffer from computational inefficiency and objective mismatches between training and inference.<n>We introduce DiRL, an efficient post-training framework that tightly integrates FlexAttention-accelerated blockwise training with LMDeploy-optimized inference.
  arXiv  Detail & Related papers  (2025-12-23T08:33:19Z)
• WAM-Flow: Parallel Coarse-to-Fine Motion Planning via Discrete Flow Matching for Autonomous Driving [9.719456684859606]
  We introduce WAM-Flow, a vision--action (VLA) model that casts ego-trajectory planning as discrete flow matching over a structured token space.<n>WAM-Flow performs fully parallel, bidirectional denoising, enabling coarse-to-fine refinement with a tunable compute-accuracy trade-off.<n>These results establish discrete flow matching as a new promising paradigm for end-to-end autonomous driving.
  arXiv  Detail & Related papers  (2025-12-05T19:36:46Z)
• AdaPtis: Reducing Pipeline Bubbles with Adaptive Pipeline Parallelism on Heterogeneous Models [59.7059443712562]
  AdaPtis is a training system for large language models (LLMs) that supports adaptive pipeline parallelism.<n>Extensive experiments show that AdaPtis achieves an average speedup of 1.42x (up to 2.14x) over Megatron-LM I-1F1B.
  arXiv  Detail & Related papers  (2025-09-28T08:05:13Z)
• CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks [57.95170323315603]
  We introduce CollaPipe, a distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving networks.<n>In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks.<n>To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power.
  arXiv  Detail & Related papers  (2025-09-24T07:54:01Z)
• Adaptive Policy Synchronization for Scalable Reinforcement Learning [0.0]
  ClusterEnv is a lightweight interface for distributed environment execution.<n>It supports both on- and off-policy methods, integrates into existing training code with minimal changes, and runs efficiently on clusters.
  arXiv  Detail & Related papers  (2025-07-15T05:07:12Z)
• AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training [24.60677187852425]
  Reinforcement learning (RL) has become a pivotal technology in the post-training phase of large language models (LLMs)<n>Traditional task-colocated RL frameworks suffer from significant scalability bottlenecks.<n>Task-separated RL frameworks face challenges in complex dataflows and the corresponding resource idling and workload imbalance.<n>We propose AsyncFlow, an asynchronous streaming RL framework for efficient post-training.
  arXiv  Detail & Related papers  (2025-07-02T12:45:34Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.