ES-Merging: Biological MLLM Merging via Embedding Space Signals

• URL: http://arxiv.org/abs/2603.14405v1
• Date: Sun, 15 Mar 2026 14:38:32 GMT
• Title: ES-Merging: Biological MLLM Merging via Embedding Space Signals
• Abstract summary: We propose a representation-aware merging framework that estimates merging coefficients from embedding space signals.<n>Our method outperforms existing merging methods and even surpasses task-specific fine-tuned models.
• Score: 52.84455878597969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Biological multimodal large language models (MLLMs) have emerged as powerful foundation models for scientific discovery. However, existing models are specialized to a single modality, limiting their ability to solve inherently cross-modal scientific problems. While model merging is an efficient method to combine the different modalities into a unified MLLM, existing methods rely on input-agnostic parameter space heuristics that fail to faithfully capture modality specialization. To overcome this limitation, we propose a representation-aware merging framework that estimates merging coefficients from embedding space signals. We first design a probe input that consists of different modality tokens and forward it through each specialized MLLM to obtain layer-wise embedding responses that reflect modality-specific representation changes. We then estimate complementary merging coefficients at two granularities from the embedding space: layer-wise coefficients from coarse-grained signals and element-wise coefficients from fine-grained signals, which are jointly combined for robust coefficient estimation. Experiments on interactive effect prediction benchmarks show that our method outperforms existing merging methods and even surpasses task-specific fine-tuned models, establishing that embedding space signals provide a principled and effective foundation for cross-modal MLLM merging.

Related papers

• Few-Shot Video Recognition via Hierarchical Metric Learning [13.999334351029196]
  Few-shot action recognition (FSAR) aims to recognize unseen action categories with only a small number of annotated video samples.<n>We present a novel method, hierarchical metric learning for few-shot action recognition (HML-FSAR)<n>The proposed HML-FSAR method is validated on five widely-used FSAR datasets, and experimental results fully demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2026-09-04T15:07:44Z)
• Influence-Guided Symbolic Regression: Scientific Discovery via LLM-Driven Equation Search with Granular Feedback [56.69850045068714]
  We introduce textitInfluence-Guided Regression (IGSR), a method that frames equation discovery as an iterative two-step process.<n>We demonstrate IGSRs effectiveness on a diverse suite of benchmarks, including LLM-SRBench, pharmacological PKPD models, an epidemiological simulation, and real-world genomic data.
  arXiv  Detail & Related papers  (2026-05-27T23:48:01Z)
• PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging [80.17966517536102]
  We argue that the core of Multimodal Large Language Models pre-training lies in establishing effective cross-modal alignment.<n>Motivated by this insight, we introduce the post-alignment merging task, which aims to integrate cross-modal alignment capabilities learned from multimodal pre-training.<n>We propose PivotMerge, a post-alignment merging framework for cross-modal projectors.
  arXiv  Detail & Related papers  (2026-04-18T09:38:03Z)
• Diffusion-CAM: Faithful Visual Explanations for dMLLMs [8.78213424930081]
  Diffusion-CAM is the first interpretability method specifically tailored for dMLLMs.<n>We derive raw activation maps by differentiably probing intermediate representations in the backbone.<n>Our experiments demonstrate that Diffusion-CAM significantly outperforms SoTA methods in both localization accuracy and visual fidelity.
  arXiv  Detail & Related papers  (2026-04-13T05:14:18Z)
• Leveraging Shared Prototypes for a Multimodal Pulse Motion Foundation Model [4.895784700544358]
  ProtoMM is a novel framework that introduces a shared prototype dictionary to anchor heterogeneous modalities in a common embedding space.<n>By clustering representations around shared prototypes rather than explicit negative sampling, our method captures complementary information across modalities and provides a coherent "common language" for physiological signals.
  arXiv  Detail & Related papers  (2025-10-10T18:13:38Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• UGD-IML: A Unified Generative Diffusion-based Framework for Constrained and Unconstrained Image Manipulation Localization [19.797719494981923]
  We propose a novel generative framework based on diffusion models, named UGD-IML, which unifies both IML and CIML tasks within a single framework.<n>We show that UGD-IML outperforms the SOTA methods by an average of 9.66 and 4.36 in terms of F1 metrics for IML and CIML tasks, respectively.
  arXiv  Detail & Related papers  (2025-08-08T08:00:28Z)
• Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs [56.76586846269894]
  Multimodal Large Language Models (MLLMs) have achieved success across various domains.<n>Despite its importance, the study of knowledge sharing among domain-specific MLLMs remains largely underexplored.<n>We propose a unified parameter integration framework that enables modular composition of expert capabilities.
  arXiv  Detail & Related papers  (2025-06-30T15:07:41Z)
• AdaMMS: Model Merging for Heterogeneous Multimodal Large Language Models with Unsupervised Coefficient Optimization [86.8133939108057]
  We propose AdaMMS, a novel model merging method tailored for heterogeneous MLLMs.<n>Our method tackles the challenges in three steps: mapping, merging and searching.<n>As the first model merging method capable of merging heterogeneous MLLMs without labeled data, AdaMMS outperforms previous model merging methods on various vision-language benchmarks.
  arXiv  Detail & Related papers  (2025-03-31T05:13:02Z)
• Reinforced Model Merging [53.84354455400038]
  We present an innovative framework termed Reinforced Model Merging (RMM), which encompasses an environment and agent tailored for merging tasks.<n>By utilizing data subsets during the evaluation process, we addressed the bottleneck in the reward feedback phase, thereby accelerating RMM by up to 100 times.
  arXiv  Detail & Related papers  (2025-03-27T08:52:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.