Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging

• URL: http://arxiv.org/abs/2605.29489v1
• Date: Thu, 28 May 2026 07:15:38 GMT
• Title: Access Sets Matter: Budgeting Expert Reads for Scalable Weight-Space Model Merging
• Abstract summary: MergePipe is a budget-aware execution layer for weight-space model merging.<n>It reduces expert-read I/O by up to an order of magnitude and achieves up to $11times$ speedups.
• Score: 29.33065526693412
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weight-space model merging is usually formulated as an algebraic operation on checkpoints, yet at LLM scale the limiting resource is often the set of expert weights that must be read. We introduce MergePipe, a budget-aware execution layer that casts LLM merging as an \emph{expert access-set} problem: given a merge operator and a checkpoint family in a shared weight coordinate system, choose which expert delta blocks to access under an explicit I/O budget. MergePipe indexes parameter blocks, builds deterministic access plans, and executes the induced budgeted merge with replayable manifests. The plan is budget-sound by construction and recovers the full-read merge at full budget; for fixed-coefficient additive operators, the omitted-update error is bounded by the norm of omitted deltas. Across Qwen and Llama merging workloads, MergePipe reduces expert-read I/O by up to an order of magnitude and achieves up to $11\times$ speedups. Representative budget sweeps show $O(10^{-3})$ parameter deviation from full-read merges and no monotonic degradation on downstream benchmarks.

Related papers

• SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling [17.873016968298483]
  Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning (PEFT) method for large language models.<n>We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in extension that re parameterizes a rank-rtot update as a sum of K static experts.
  arXiv  Detail & Related papers  (2026-06-27T03:25:36Z)
• Compress then Merge: From Multiple LoRAs into One Low-Rank Adapter [31.066992554963093]
  We study the problem of merging $T$ LoRAs into a single rank-$r$ LoRA.<n>Existing-then-Compress pipelines treat the rank constraint as an afterthought.<n>We propose Compress-then-Merge (CtM), a reversed pipeline that enforces the rank-$r$ bottleneck before merging.
  arXiv  Detail & Related papers  (2026-06-02T14:42:46Z)
• ZEBRA: Zero-shot Budgeted Resource Allocation for LLM Orchestration [8.226365534099399]
  We propose ZEBRA, a framework that reduces multi-phase budget allocation to a continuous nonlinear knapsack problem.<n>On a $150$-task APPS coding benchmark, both ZEBRA variants outperform LLM-direct on every aggregate metric.<n>We show that lightweight algorithmic guidance at inference time can improve the economic behavior of autonomous multi-agent systems.
  arXiv  Detail & Related papers  (2026-05-19T20:50:05Z)
• Evolutionary Negative Module Pruning for Better LoRA Merging [46.397675691417675]
  Merging multiple Low-Rank Adaptation experts into a single backbone is a promising approach for efficient multi-task deployment.<n>Existing methods strive to alleviate interference via weight or subspace alignment.<n>We propose a plug-and-play LoRA pruning method to locate and exclude these modules prior to merging.
  arXiv  Detail & Related papers  (2026-04-20T03:13:18Z)
• Label-Free Cross-Task LoRA Merging with Null-Space Compression [50.63908869296697]
  We introduce Null-Space Compression (NSC) Merging, a label-free, output-agnostic method that sets merge weights from adapter geometry.<n>NSC achieves state-of-the-art performance across twenty heterogeneous vision tasks with balanced gains where prior methods overfit subsets of tasks.
  arXiv  Detail & Related papers  (2026-03-27T11:34:41Z)
• MergePipe: A Budget-Aware Parameter Management System for Scalable LLM Merging [37.422249601576965]
  MergePipe is a parameter management system for scalable LLM merging.<n>It employs a cost-aware planner that explicitly models expert parameter I/O and enforces user-specified I/O budgets.<n>Experiments show that MergePipe reduces total I/O by up to an order of magnitude and delivers up to $11times$ end-to-end speedups.
  arXiv  Detail & Related papers  (2026-02-05T05:06:11Z)
• Multiple-play Stochastic Bandits with Prioritized Arm Capacity Sharing [52.124267908936396]
  The model is composed of $M$ arms and $K$ plays.<n>Each arm has a number of capacities, and each unit of capacity is associated with a reward function.<n>When multiple plays compete for the arm capacity, the arm capacity is allocated in a larger priority weight first manner.
  arXiv  Detail & Related papers  (2025-12-25T11:19:09Z)
• Rethinking Parameter Sharing for LLM Fine-Tuning with Multiple LoRAs [26.212332132619736]
  We propose an asymmetric multi-LoRA design with multiple $A$ matrices and a single shared $B$ in multi-task fine-tuning.<n>Our methods achieve more balanced performance across tasks with comparable or superior average accuracy relative to existing multi-LoRA approaches.
  arXiv  Detail & Related papers  (2025-09-29T19:16:14Z)
• The Alignment Bottleneck [0.0]
  We model the loop as a two-stage cascade $U to H to Y$ given $S$, with cognitive capacity $C_textcog|S$ and average total capacity $barC_texttot|S$.<n>It pairs a data size-independent Fano lower bound proved on a separable codebook mixture with a PAC-Bayes upper bound whose KL term is controlled by the same channel via $m, barC_texttot|S$.
  arXiv  Detail & Related papers  (2025-09-19T12:38:30Z)
• Drag-and-Drop LLMs: Zero-Shot Prompt-to-Weights [75.83625828306839]
  textbfDrag-and-Drop LLMs (textitDnD) eliminates per-task training by mapping a handful of unlabeled task prompts directly to LoRA weight updates.<n>A lightweight text encoder distills each prompt batch into condition embeddings, which are then transformed by a cascaded hyper-convolutional decoder into the full set of LoRA matrices.
  arXiv  Detail & Related papers  (2025-06-19T15:38:21Z)
• Near-Optimal Online Learning for Multi-Agent Submodular Coordination: Tight Approximation and Communication Efficiency [52.60557300927007]
  We present a $textbfMA-OSMA$ algorithm to transfer the discrete submodular problem into a continuous optimization.<n>We also introduce a projection-free $textbfMA-OSEA$ algorithm, which effectively utilizes the KL divergence by mixing a uniform distribution.<n>Our algorithms significantly improve the $(frac11+c)$-approximation provided by the state-of-the-art OSG algorithm.
  arXiv  Detail & Related papers  (2025-02-07T15:57:56Z)
• Combinatorial Stochastic-Greedy Bandit [79.1700188160944]
  We propose a novelgreedy bandit (SGB) algorithm for multi-armed bandit problems when no extra information other than the joint reward of the selected set of $n$ arms at each time $tin [T]$ is observed. SGB adopts an optimized-explore-then-commit approach and is specifically designed for scenarios with a large set of base arms.
  arXiv  Detail & Related papers  (2023-12-13T11:08:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.