LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems

• URL: http://arxiv.org/abs/2507.21276v1
• Date: Mon, 28 Jul 2025 19:03:26 GMT
• Title: LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systems
• Abstract summary: We propose LeMix, a system for co-locating and managing concurrent large language models (LLMs) serving and training workloads.<n>LeMix integrates offline profiling, execution prediction mechanisms, and runtime scheduling to dynamically adapt resource allocation.<n>Our evaluation shows that LeMix improves throughput by up to 3.53x, reduces inference loss by up to 0.61x, and delivers up to 2.12x higher response time SLO attainment.
• Score: 20.108099254767613
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modern deployment of large language models (LLMs) frequently involves both inference serving and continuous retraining to stay aligned with evolving data and user feedback. Common practices separate these workloads onto distinct servers in isolated phases, causing substantial inefficiencies (e.g., GPU idleness) and delayed adaptation to new data in distributed settings. Our empirical analysis reveals that these inefficiencies stem from dynamic request arrivals during serving and workload heterogeneity in pipeline-parallel training. To address these challenges, we propose LeMix, a system for co-locating and managing concurrent LLM serving and training workloads. LeMix integrates offline profiling, execution prediction mechanisms, and runtime scheduling to dynamically adapt resource allocation based on workload characteristics and system conditions. By understanding task-specific behaviors and co-execution interference across shared nodes, LeMix improves utilization and serving quality without compromising serving responsiveness. Our evaluation shows that LeMix improves throughput by up to 3.53x, reduces inference loss by up to 0.61x, and delivers up to 2.12x higher response time SLO attainment over traditional separate setups. To our knowledge, this is the first work to uncover and exploit the opportunities of joint LLM inference and training, paving the way for more resource-efficient deployment of LLMs in production environments.

Related papers

• FlowTrain: Flow-Based Decoupled Training for Industrial-Grade Vision-Language Models [31.501910341506164]
  FlowTrain is a flow-based decoupled training framework for vision-language computation models.<n>It reformulates VLM training as a producer-consumer dataflow coordinated through a unified memory pool.<n>Experiments on real-world workloads show that FlowTrain achieves over 50% MFU and up to 1.7x throughput improvement.
  arXiv  Detail & Related papers  (2026-06-22T09:33:44Z)
• MOSAIC: Efficient Mixture-of-Agent Scheduling via Adaptive Aggregation and Inference Concurrency [11.75073577935001]
  Mixture-of-Agents (MoA) systems improve reasoning accuracy by routing each query to multiple expert LLMs and aggregating their outputs.<n>Traditional scheduling strategies suffer from significant GPU idling and throughput collapse due to load imbalances.<n>We present MOSAIC, a scheduling framework to accelerate MoA workloads on limited GPU resources.
  arXiv  Detail & Related papers  (2026-06-02T01:40:33Z)
• CoLLM: A Unified Framework for Co-execution of LLMs Federated Fine-tuning and Inference [33.14453252489961]
  CoLLM is a system that unifies FL PEFT and inference on shared edge replicas and model parameters.<n>It consistently outperforms state-of-the-art LLM systems, achieving up to 3x higher goodput.
  arXiv  Detail & Related papers  (2026-03-31T09:49:47Z)
• Rollout-Training Co-Design for Efficient LLM-Based Multi-Agent Reinforcement Learning [21.32826098306222]
  Existing training frameworks fail to address the unique system-level challenges of multi-agent reinforcement learning.<n>We propose FlexMARL, the first end-to-end training framework that holistically optimize rollout, training, and their orchestration.<n>We show that FlexMARL achieves up to 7.3x speedup and improves hardware utilization by up to 5.6x compared to existing frameworks.
  arXiv  Detail & Related papers  (2026-02-10T09:27:03Z)
• MACE: A Hybrid LLM Serving System with Colocated SLO-aware Continuous Retraining Alignment [14.392166280035122]
  Large language models (LLMs) deployed on edge servers are increasingly used in latency-sensitive applications such as personalized assistants, recommendation, and content moderation.<n>Existing retraining strategies either delay model updates, over-commit resources to retraining, or overlook iteration-level retraining granularity.<n>We propose MACE, a hybrid LLM system that colocates concurrent inference (prefill, decode) and fine-tuning, with intelligent memory management to maximize task performance while promising inference throughput.
  arXiv  Detail & Related papers  (2025-09-28T18:45:28Z)
• CollaPipe: Adaptive Segment-Optimized Pipeline Parallelism for Collaborative LLM Training in Heterogeneous Edge Networks [57.95170323315603]
  We introduce CollaPipe, a distributed learning framework that integrates collaborative pipeline parallelism with federated aggregation to support self-evolving networks.<n>In CollaPipe, the encoder part is adaptively partitioned into variable-sized segments and deployed across mobile devices for pipeline-parallel training, while the decoder is deployed on edge servers to handle generative tasks.<n>To enhance training efficiency, we formulate a joint optimization problem that adaptively allocates model segments, micro-batches, bandwidth, and transmission power.
  arXiv  Detail & Related papers  (2025-09-24T07:54:01Z)
• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism [9.93378263858092]
  Multimodal large language models (MLLMs) handle images, videos, and audio by incorporating feature extractors and projection modules.<n>Current tightly coupled serving architectures struggle to distinguish between mixed request types.<n>We propose Elastic Multimodal Parallelism (EMP), a new serving paradigm that elastically adapts to resource heterogeneity.
  arXiv  Detail & Related papers  (2025-07-14T08:53:48Z)
• PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMs [56.04036826558497]
  We introduce Privacy-preserving Collaborative Mixture-of-Experts (PC-MoE)<n>By design, PC-MoE synergistically combines the strengths of distributed computation with strong confidentiality assurances.<n>It almost matches (and sometimes exceeds) the performance and convergence rate of a fully centralized model, enjoys near 70% peak GPU RAM reduction, while being fully robust against reconstruction attacks.
  arXiv  Detail & Related papers  (2025-06-03T15:00:18Z)
• LLMPrism: Black-box Performance Diagnosis for Production LLM Training Platforms [31.576014566773697]
  Large Language Models (LLMs) have brought about revolutionary changes in diverse fields.<n>This paper proposes the utilization of underlying network flow data to reconstruct the training timelines of jobs.<n>We design LLMPrism, the first black-box performance diagnosis system for LLM training platforms.
  arXiv  Detail & Related papers  (2025-05-01T06:38:52Z)
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation [55.75008325187133]
  Reinforcement learning (RL) has become the core post-training technique for large language models (LLMs)<n>StreamRL is designed with disaggregation from first principles to address two types of performance bottlenecks.<n> Experiments show that StreamRL improves throughput by up to 2.66x compared to existing state-of-the-art systems.
  arXiv  Detail & Related papers  (2025-04-22T14:19:06Z)
• MixLLM: Dynamic Routing in Mixed Large Language Models [57.309520357563215]
  Large Language Models (LLMs) exhibit potential artificial generic intelligence recently, however, their usage is costly with high response latency.<n>We develop MixLLM, a dynamic contextual-bandit-based routing system for query-LLM assignment.
  arXiv  Detail & Related papers  (2025-02-09T02:26:15Z)
• LLM-Based Offline Learning for Embodied Agents via Consistency-Guided Reward Ensemble [4.41983632543407]
  We present a consistency-guided reward ensemble framework (CoREN) for embodied agent learning. The framework employs an adaptive ensemble oftemporally consistent rewards to derive domain-grounded rewards in the training datasets.
  arXiv  Detail & Related papers  (2024-11-26T06:04:10Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs [74.35290684163718]
  A primary challenge in large language model (LLM) development is their onerous pre-training cost. This paper explores a promising paradigm to improve LLM pre-training efficiency and quality by leveraging a small language model (SLM)
  arXiv  Detail & Related papers  (2024-10-24T14:31:52Z)
• Efficient Continual Pre-training by Mitigating the Stability Gap [68.49269649759005]
  We study the behavior of Large Language Models (LLMs) during continual pre-training. We propose three effective strategies to enhance LLM performance within a fixed compute budget. Our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% with only 40% of the original training budget.
  arXiv  Detail & Related papers  (2024-06-21T02:28:37Z)
• One QuantLLM for ALL: Fine-tuning Quantized LLMs Once for Efficient Deployments [43.107261545706415]
  Large Language Models (LLMs) have advanced rapidly but face significant memory demands. Current methods typically require lengthy training to alleviate the performance degradation from quantization loss. We make an initial attempt to extend the once-for-all framework to large language models.
  arXiv  Detail & Related papers  (2024-05-30T16:05:15Z)
• Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction [8.705908108054878]
  Large models (LLMs) have been driving a new wave of AI applications across numerous domains. We present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths.
  arXiv  Detail & Related papers  (2024-04-12T14:46:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.