Can Heterogeneous Language Models Be Fused?

• URL: http://arxiv.org/abs/2604.01674v1
• Date: Thu, 02 Apr 2026 06:21:21 GMT
• Title: Can Heterogeneous Language Models Be Fused?
• Abstract summary: Model merging aims to integrate multiple expert models into a single model that inherits their complementary strengths without incurring the inference-time cost of ensembling.<n>Recent progress has shown that merging can be highly effective when all source models are emphhomogeneous, i.e., derived from the same pretrained backbone.<n>Yet this assumption is increasingly unrealistic in open model ecosystems, where useful experts are often built on different families such as Llama, Qwen, and Mistral.<n>In such emphheterogeneous settings, direct weight-space fusion becomes ill-posed due
• Score: 24.088997528611873
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model merging aims to integrate multiple expert models into a single model that inherits their complementary strengths without incurring the inference-time cost of ensembling. Recent progress has shown that merging can be highly effective when all source models are \emph{homogeneous}, i.e., derived from the same pretrained backbone and therefore share aligned parameter coordinates or compatible task vectors. Yet this assumption is increasingly unrealistic in open model ecosystems, where useful experts are often built on different families such as Llama, Qwen, and Mistral. In such \emph{heterogeneous} settings, direct weight-space fusion becomes ill-posed due to architectural mismatch, latent basis misalignment, and amplified cross-source conflict. We address this problem with \texttt{HeteroFusion} for heterogeneous language model fusion, which consists of two key components: topology-based alignment that transfers knowledge across heterogeneous backbones by matching functional module structures instead of raw tensor coordinates, and conflict-aware denoising that suppresses incompatible or noisy transfer signals during fusion. We further provide analytical justification showing that preserving the target adapter basis while predicting structured updates leads to a stable and well-conditioned transfer process. Across heterogeneous transfer, multi-source fusion, noisy-source robustness, and cross-family generalization settings, \texttt{HeteroFusion} consistently outperforms strong merging, fusion, and ensemble baselines.

Related papers

• Impute-EM: Native Mixed-State Diffusion Models for Heterogeneous Data Imputation [71.74203645336385]
  Impute-EM alternates between imputing missing entries with the current model and refitting a diffusion backbone on completed data.<n>We characterize the update and show that the observed mask-indexed marginals match the targets at the limit.
  arXiv  Detail & Related papers  (2026-09-14T09:39:13Z)
• Understanding Synergistic Interactions among Pathology Foundation Models via Adaptive Fusion [49.550545038402184]
  Pathology foundation models (PFMs) provide strong tile-level representations via self-supervised pre-training on large-scale pathology images.<n>We present AdaFusion, a lightweight adaptive fusion framework that integrates complementary signals from multiple frozen PFMs.<n>AdaFusion consistently outperforms individual PFMs and other fusion baselines, while providing interpretable tissue visualisation.
  arXiv  Detail & Related papers  (2026-08-02T16:44:45Z)
• Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients [22.724544289853558]
  FedSLM is a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients.<n>We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise.
  arXiv  Detail & Related papers  (2026-07-31T06:45:56Z)
• Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective [41.30735431552607]
  heterogeneous fusion methods typically introduce adapters, learned latent spaces, routing, or feature alignment.<n>We revisit this counterintuitive question through training-free dimensional adaptation followed by ratio-controlled ratios.<n>Results show that simple parameter averaging, when paired with lightweight dimensional adaptation and carefully controlled ratios, is a surprisingly strong baseline for heterogeneous merging.
  arXiv  Detail & Related papers  (2026-07-20T14:58:53Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• Empowering Heterogeneous Graph Foundation Models via Decoupled Relation Alignment [20.128308877213044]
  Decoupled relation Subspace Alignment (DRSA) is a novel, plug-and-play relation-driven framework.<n>It fundamentally shifts the paradigm by decoupling feature semantics from relation structures.<n>It can be seamlessly integrated as a universal preprocessing module.
  arXiv  Detail & Related papers  (2026-05-01T15:31:36Z)
• Generative Data Transformation: From Mixed to Unified Data [57.84692191369066]
  textscTaesar is a emphdata-centric framework for textbftarget-textbfal textbfregeneration.<n>It encodes cross-domain context into target sequences, enabling standard models to learn intricate dependencies without complex fusion architectures.
  arXiv  Detail & Related papers  (2026-02-26T08:30:09Z)
• Annealed Co-Generation: Disentangling Variables via Progressive Pairwise Modeling [29.29369402735124]
  We propose an Annealed Co-Generation framework that replaces high-dimensional diffusion modeling with a low-dimensional diffusion model.<n>We demonstrate the framework's flexibility and efficacy on two distinct scientific tasks.
  arXiv  Detail & Related papers  (2026-02-20T07:42:11Z)
• Breaking the Factorization Barrier in Diffusion Language Models [59.946071582340146]
  "Factorization barrier" impedes efficient parallel generation of diffusion language models.<n>We propose Coupled Discrete Diffusion to replace the fully-factorized output distribution.<n>We show that CoDD seamlessly enhances diverse diffusion language model architectures with negligible overhead.
  arXiv  Detail & Related papers  (2026-02-09T08:36:39Z)
• Disentangling History and Propagation Dependencies in Cross-Subject Knee Contact Stress Prediction Using a Shared MeshGraphNet Backbone [0.8283940114367679]
  It remains unclear whether the dominant source of prediction uncertainty arises from temporal history dependence or spatial propagation dependence.<n>A dataset of running trials from nine subjects was constructed using an OpenSim-FEBio workflow.<n>Models incorporating history encoding significantly outperformed the baseline MGN and ModMGN in global accuracy and spatial consistency.
  arXiv  Detail & Related papers  (2026-01-13T08:15:57Z)
• Bridging the Discrete-Continuous Gap: Unified Multimodal Generation via Coupled Manifold Discrete Absorbing Diffusion [60.186310080523135]
  Bifurcation of generative modeling into autoregressive approaches for discrete data (text) and diffusion approaches for continuous data (images) hinders development of truly unified multimodal systems.<n>We propose textbfCoM-DAD, a novel probabilistic framework that reformulates multimodal generation as a hierarchical dual-process.<n>Our method demonstrates superior stability over standard masked modeling, establishing a new paradigm for scalable, unified text-image generation.
  arXiv  Detail & Related papers  (2026-01-07T16:21:19Z)
• The Trojan in the Vocabulary: Stealthy Sabotage of LLM Composition [31.827344197678126]
  Tokenizer transplant introduces a supply-chain vulnerability.<n>By exploiting the geometry of coefficient reuse, our attack creates an asymmetric realizability gap.<n> Empirically, the attack is training-free and achieves spectral mimicry to evade outlier detection.
  arXiv  Detail & Related papers  (2025-12-31T19:00:03Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• H2Tune: Federated Foundation Model Fine-Tuning with Hybrid Heterogeneity [25.639893635287187]
  We propose H2Tune, a federated foundation model fine-tuning with hybrid heterogeneity.<n>Our framework consists of three key components: (i) sparsified triple matrix decomposition to align hidden dimensions across clients through constructing rank-consistent middle matrices, with adaptive sparsification based on client resources; (ii) relation-guided matrix layer alignment to handle heterogeneous layer structures and representation capabilities; and (iii) alternating task-knowledge disentanglement mechanism to decouple shared and specific knowledge of local model parameters through alternating optimization.
  arXiv  Detail & Related papers  (2025-07-30T12:53:18Z)
• Spatial Semantic Recurrent Mining for Referring Image Segmentation [63.34997546393106]
  We propose Stextsuperscript2RM to achieve high-quality cross-modality fusion. It follows a working strategy of trilogy: distributing language feature, spatial semantic recurrent coparsing, and parsed-semantic balancing. Our proposed method performs favorably against other state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2024-05-15T00:17:48Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.