MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging

• URL: http://arxiv.org/abs/2602.13273v1
• Date: Thu, 05 Feb 2026 05:06:11 GMT
• Title: MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging
• Abstract summary: MergePipe is a parameter management system for scalable LLM merging.<n>It employs a cost-aware planner that explicitly models expert parameter I/O and enforces user-specified I/O budgets.<n>Experiments show that MergePipe reduces total I/O by up to an order of magnitude and delivers up to $11times$ end-to-end speedups.
• Score: 37.422249601576965
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) merging has become a key technique in modern LLM development pipelines, enabling the integration of multiple task- or domain-specific expert models without retraining. However, as the number of experts grows, existing merging implementations treat model parameters as unstructured files and execute merges in a stateless, one-shot manner, leading to excessive disk I/O, redundant parameter scans, and poor scalability. In this paper, we present \textbf{MergePipe}, a parameter management system for scalable LLM merging. MergePipe is the first system that treats LLM merging as a data management and execution problem, and introduces a catalog-driven abstraction over model parameters, merge plans, and execution lineage. At its core, MergePipe employs a cost-aware planner that explicitly models expert parameter I/O and enforces user-specified I/O budgets, followed by a streaming execution engine that materializes merged models under transactional guarantees. Our key insight is that while base model reads and output writes are unavoidable, expert parameter reads dominate merge cost and constitute the primary optimization target. By making expert access budget-aware throughout planning and execution, MergePipe mitigates the $O(K)$ I/O growth of naive pipelines and achieves predictable scaling behavior. Experiments show that MergePipe reduces total I/O by up to an order of magnitude and delivers up to $11\times$ end-to-end speedups (up to 90\% wall-time reduction) over state-of-the-art LLM merging pipelines.

Related papers

• GRIMIP: A General Framework for Instance-Specific Configuration of MIP Solvers Using LLMs [9.2939109839777]
  This paper introduces textbfGRIMIP (textbfunderlineGeneral textbfunderlineReasoning for textbfunderlineInstance-specific textbfunderlineMIP configuration), a novel hybrid intelligence framework.
  arXiv  Detail & Related papers  (2026-06-22T13:10:50Z)
• Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging [29.33065526693412]
  MergePipe is a budget-aware execution layer for weight-space model merging.<n>It reduces expert-read I/O by up to an order of magnitude and achieves up to $11times$ speedups.
  arXiv  Detail & Related papers  (2026-05-28T07:15:38Z)
• DiffuRank: Effective Document Reranking with Diffusion Language Models [71.16830004674513]
  We propose DiffuRank, a reranking framework built upon diffusion language models (dLLMs)<n>dLLMs support more flexible decoding and generation processes that are not constrained to a left-to-right order.<n>We show dLLMs achieve performance comparable to, and in some cases exceeding, that of autoregressive LLMs with similar model sizes.
  arXiv  Detail & Related papers  (2026-02-13T02:18:14Z)
• The Case for Instance-Optimized LLMs in OLAP Databases [0.7090165638014332]
  Large Language Models (LLMs) can enhance analytics systems with powerful data summarization, cleaning, and semantic transformation capabilities.<n>We present IOLMDB, a novel system that makes LLM-enhanced database queries practical through query-specific model optimization.
  arXiv  Detail & Related papers  (2025-07-07T13:10:01Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• Multi-Modality Expansion and Retention for LLMs through Parameter Merging and Decoupling [33.90190836855285]
  MMER integrates existing MLLMs for effective multimodal expansion while retaining their original performance.<n>MMER can also mitigate catastrophic forgetting by applying a similar process to MLLMs fine-tuned on new tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:40:07Z)
• AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization [86.8133939108057]
  We propose AdaMMS, a novel model merging method tailored for heterogeneous MLLMs.<n>Our method tackles the challenges in three steps: mapping, merging and searching.<n>As the first model merging method capable of merging heterogeneous MLLMs without labeled data, AdaMMS outperforms previous model merging methods on various vision-language benchmarks.
  arXiv  Detail & Related papers  (2025-03-31T05:13:02Z)
• Reinforced Model Merging [53.84354455400038]
  We present an innovative framework termed Reinforced Model Merging (RMM), which encompasses an environment and agent tailored for merging tasks.<n>By utilizing data subsets during the evaluation process, we addressed the bottleneck in the reward feedback phase, thereby accelerating RMM by up to 100 times.
  arXiv  Detail & Related papers  (2025-03-27T08:52:41Z)
• Mixup Model Merge: Enhancing Model Merging Performance through Randomized Linear Interpolation [15.47711837051754]
  Model merging aims to integrate multiple task-specific models into a unified model that inherits the capabilities of the task-specific models.<n>Existing model merging methods often lack consideration of the varying contribution ratios of different task-specific models to the final merged model.<n>We propose Mixup Model Merge (M3), a simple yet effective method inspired by the randomized linear strategy from the Mixup data augmentation technique.
  arXiv  Detail & Related papers  (2025-02-21T13:01:26Z)
• Fine, I'll Merge It Myself: A Multi-Fidelity Framework for Automated Model Merging [30.38047100067552]
  Reasoning capabilities represent a critical frontier for large language models.<n>One way to efficiently supplement capabilities with is by model merging.<n>We propose an Automated Model Merging Framework that enables fine-grained exploration of merging strategies.
  arXiv  Detail & Related papers  (2025-02-06T12:47:25Z)
• Channel Merging: Preserving Specialization for Merged Experts [16.122289324737366]
  We introduce Channel Merging, a novel strategy designed to minimize parameter conflicts while enhancing storage efficiency.<n>Our experiments demonstrate that Channel Merging consistently delivers high performance, matching unmerged models in tasks like English and Chinese reasoning, mathematical reasoning, and code generation.
  arXiv  Detail & Related papers  (2024-12-18T16:07:44Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline.<n>Recent works have started exploiting large language models (LLM) to lessen such burden.<n>This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction [8.705908108054878]
  Large models (LLMs) have been driving a new wave of AI applications across numerous domains. We present a speculative shortest-job-first (SSJF) scheduler that uses a light proxy model to predict LLM output sequence lengths.
  arXiv  Detail & Related papers  (2024-04-12T14:46:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.