Transport and Merge: Cross-Architecture Merging for Large Language Models

• URL: http://arxiv.org/abs/2602.05495v1
• Date: Thu, 05 Feb 2026 09:57:57 GMT
• Title: Transport and Merge: Cross-Architecture Merging for Large Language Models
• Authors: Chenhang Cui, Binyun Yang, Fei Shen, Yuxin Chen, Jingnan Zheng, Xiang Wang, An Zhang, Tat-Seng Chua,
• Abstract summary: Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data.<n>Many real-world deployments rely on smaller models trained or adapted from low-resource data.<n>This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets.
• Score: 59.53629883788284
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) achieve strong capabilities by scaling model capacity and training data, yet many real-world deployments rely on smaller models trained or adapted from low-resource data. This gap motivates the need for mechanisms to transfer knowledge from large, high-resource models to smaller, low-resource targets. While model merging provides an effective transfer mechanism, most existing approaches assume architecture-compatible models and therefore cannot directly transfer knowledge from large high-resource LLMs to heterogeneous low-resource targets. In this work, we propose a cross-architecture merging framework based on optimal transport (OT) that aligns activations to infer cross-neuron correspondences between heterogeneous models. The resulting transport plans are then used to guide direct weight-space fusion, enabling effective high-resource to low-resource transfer using only a small set of inputs. Extensive experiments across low-resource languages and specialized domains demonstrate consistent improvements over target models.

Related papers

• AutoMerge: Search-Based Model Merging Framework for Effective Model Reuse [8.950520457150178]
  Recently, model merging has emerged in the domain of large language models (LLMs) as a training-free approach.<n>No prior work has systematically investigated whether such an approach can be effectively applied to other deep learning models.<n>We present the first systematic study that evaluates five model merging techniques on three distinct model architectures.
  arXiv  Detail & Related papers  (2026-01-30T09:27:01Z)
• NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matching [64.10695425442164]
  We introduce NExT-OMNI, an open-source omnimodal foundation model that achieves unified modeling through discrete flow paradigms.<n>Trained on large-scale interleaved text, image, video, and audio data, NExT-OMNI delivers competitive performance on multimodal generation and understanding benchmarks.<n>To advance further research, we release training details, data protocols, and open-source both the code and model checkpoints.
  arXiv  Detail & Related papers  (2025-10-15T16:25:18Z)
• Speed Always Wins: A Survey on Efficient Architectures for Large Language Models [51.817121227562964]
  Large Language Models (LLMs) have delivered impressive results in language understanding, generation, reasoning, and pushes the ability boundary of multimodal models.<n> Transformer models, as the foundation of modern LLMs, offer a strong baseline with excellent scaling properties.<n>The traditional transformer architecture requires substantial computations and poses significant obstacles for large-scale training and practical deployment.
  arXiv  Detail & Related papers  (2025-08-13T14:13:46Z)
• OptMerge: Unifying Multimodal LLM Capabilities and Modalities via Model Merging [124.91183814854126]
  Model merging seeks to combine multiple expert models into a single model.<n>We introduce a benchmark for model merging research that clearly divides the tasks for MLLM training and evaluation.<n>We find that model merging offers a promising way for building improved MLLMs without requiring training data.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• LatentLLM: Attention-Aware Joint Tensor Compression [50.33925662486034]
  Large language models (LLMs) and large multi-modal models (LMMs) require a massive amount of computational and memory resources.<n>We propose a new framework to convert such LLMs/LMMs into a reduced-dimension latent structure.
  arXiv  Detail & Related papers  (2025-05-23T22:39:54Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression [64.07696663255155]
  Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. We introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT) which effectively transfers the knowledge of LLMs to extremely small-scale models.
  arXiv  Detail & Related papers  (2023-10-24T07:58:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.