Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training

• URL: http://arxiv.org/abs/2605.26484v1
• Date: Tue, 26 May 2026 02:48:34 GMT
• Title: Extra-Merge: Tracing the Rank-1 Subspace of Model Merging in Language Model Pre-Training
• Abstract summary: Extra-Merge consistently outperforms standard merging baselines in experiments across GPT-2 and LLaMA families.<n>It yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon citepjordan2024muon.
• Score: 56.323119575322146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model merging has emerged as a lightweight paradigm for enhancing Large Language Models (LLMs), yet its underlying mechanisms remain poorly understood. In this work, we analyze late-stage pre-training trajectories and uncover a \textbf{Rank-1 Subspace} phenomenon: while raw optimization steps oscillate violently, consecutive \emph{merged} checkpoints collapse onto a stable, approximately one-dimensional linear manifold. We theoretically ground this observation in a \emph{river-valley} landscape analysis: averaging acts as a geometric low-pass filter that dampens high-curvature noise to reveal the optimal descent direction. Capitalizing on this insight, we propose \textbf{Extra-Merge}, a training-free strategy that extrapolates along this subspace to minimize loss without additional gradient updates. Extensive experiments across GPT-2 and LLaMA families (124M to 2B) demonstrate that Extra-Merge consistently outperforms standard merging baselines. Notably, it yields consistent zero-shot accuracy gains on Pythia-12B downstream tasks and generalizes effectively to the Muon optimizer \citep{jordan2024muon}.

Related papers

• Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average [0.8224695424591678]
  emphAMFmbox-VImbox-sEMA is a two-stage framework featuring a emphstable global weighting mechanism.<n>It achieves consistent NLL improvements over its predecessor emphAMFmbox-VI and avoids the catastrophic transport failures of single-flow baselines.
  arXiv  Detail & Related papers  (2026-07-04T10:31:10Z)
• The Multiscale Single-Index Model: A Stylized Model for Hierarchical Feature Learning [33.59654714700262]
  We consider the Multiscale Single-Index Model (MSIM) as a stylized model for hierarchical learning with emphscale separation.<n>We perform a fine-grained analysis of the chaos using Wiener Edgeworth expansions.<n>We prove that online SGD on the correlation objective, where all layers evolve in the same timescale, $1 - o_d(1)$ recovery with $n = widetildeO( dK-1)$ samples, recovering the same sample complexity as in the linear counterpart.
  arXiv  Detail & Related papers  (2026-07-03T14:02:42Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• Efficient Adjoint Matching for Fine-tuning Diffusion Models [47.05085426361986]
  We propose EAM, which substantially improves training efficiency by reformulating the SOC problem with a textit base drift and a textitterminal cost.<n>EAM converges up to 4x faster than AM and matches or surpasses it across various metrics including PickScore, ImageReward, HPSv2.1, CLIPScore and Aesthetics.
  arXiv  Detail & Related papers  (2026-05-12T03:55:12Z)
• Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning [72.62839712454196]
  Recent advances in spectral optimization, notably Muon, have demonstrated that constraining update steps to the Stiefel manifold can significantly accelerate training and improveimation.<n>We argue that this "egalitarian" constraint is suboptimal for Deep Neural Networks, where the curvature spectrum is known to be highly heavy-tailed and ill-conditioned.<n>We propose textbfMousse (textbfMuon textbfOptimization textbfUtilizing textbfShampoo's textbfStruct
  arXiv  Detail & Related papers  (2026-03-10T14:03:49Z)
• Robust and Efficient Zeroth-Order LLM Fine-Tuning via Adaptive Bayesian Subspace Optimizer [4.6561758107970395]
  Fine-tuning large language models (LLMs) with zeroth-order (ZO) optimization reduces memory by approximating gradients through function evaluations.<n>We introduce BSZO, an adaptive textbfBayesian textbfSubspace textbfZeroth-Order textbfOptimizer.
  arXiv  Detail & Related papers  (2026-01-04T09:35:11Z)
• MaP: A Unified Framework for Reliable Evaluation of Pre-training Dynamics [72.00014675808228]
  Instability in Large Language Models evaluation process obscures true learning dynamics.<n>We introduce textbfMaP, a framework that integrates underlineMerging underlineand the underlinePass@k metric.<n>Experiments show that MaP yields significantly smoother performance curves, reduces inter-run variance, and ensures more consistent rankings.
  arXiv  Detail & Related papers  (2025-10-10T11:40:27Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Self-Boost via Optimal Retraining: An Analysis via Approximate Message Passing [58.52119063742121]
  Retraining a model using its own predictions together with the original, potentially noisy labels is a well-known strategy for improving the model performance.<n>This paper addresses the question of how to optimally combine the model's predictions and the provided labels.<n>Our main contribution is the derivation of the Bayes optimal aggregator function to combine the current model's predictions and the given labels.
  arXiv  Detail & Related papers  (2025-05-21T07:16:44Z)
• Deep Momentum Multi-Marginal Schr\"odinger Bridge [41.27274841596343]
  We present a novel framework that learns the smooth measure-valued algorithm for systems that satisfy position marginal constraints across time. Our algorithm outperforms baselines significantly, as evidenced by experiments for synthetic datasets and a real-world single-cell RNA dataset sequence.
  arXiv  Detail & Related papers  (2023-03-03T07:24:38Z)
• Interpolating Discriminant Functions in High-Dimensional Gaussian Latent Mixtures [1.4213973379473654]
  This paper considers binary classification of high-dimensional features under a postulated model. A generalized least squares estimator is used to estimate the direction of the optimal separating hyperplane.
  arXiv  Detail & Related papers  (2022-10-25T21:19:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.