Curvature-Guided Mixing for MLLM Adaptation

• URL: http://arxiv.org/abs/2606.24963v1
• Date: Tue, 23 Jun 2026 09:21:54 GMT
• Title: Curvature-Guided Mixing for MLLM Adaptation
• Abstract summary: We propose Curvature Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models.<n>CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio.<n>We also introduce CGM$dagger$, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score.
• Score: 13.796698666991938
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning Multimodal Large Language Models (MLLMs) on specialized tasks often leads to catastrophic forgetting of their general capabilities. Existing model merging methods to combat this are often heuristic or use sub-optimal objectives. We propose CurvatureGuided Mixing (CGM), a theoretically grounded framework that merges pre-trained and fine-tuned models. CGM formulates a joint optimization objective and uses a second-order (Hessian) approximation of the loss landscapes to analytically derive an optimal, closed-form "soft mixing" ratio. This ratio intelligently blends parameters based on their relative task-specific curvatures. We also introduce CGM$\dagger$, a robust "hard mixing" variant that performs sparse parameter selection guided by a novel, curvature-aware score. Experiments on LLaVA-1.5 and Qwen2.5VL across multiple downstream tasks show that CGM and CGM$\dagger$ consistently improve the trade-off between task specialization and general knowledge retention over existing methods. Code is available at github.com/zzsyjl/CGM-ECCV-2026.

Related papers

• CoMerge: Conflict-Driven Preference Optimization for Multi-Task Model Merging [59.836482609414816]
  We propose a conflict-driven preference optimization framework for model merging (CoMerge)<n>CoMerge enables the model to mitigate parameter-space conflicts while preserving task-specific capabilities.<n>Experiments show that CoMerge achieves an average normalized performance of 0.9968 on MergeBench.
  arXiv  Detail & Related papers  (2026-09-02T08:21:28Z)
• Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection [85.49213290363834]
  We propose an Understanding-Enhanced Model Collaboration Method (UE-MCM)<n>It combines efficient coarse-grained video understanding with accurate fine-grained action reasoning.<n>The resulting system balances speed and accuracy, making it effective for detecting subtle, rare, and ambiguous mistakes in egocentric instructional videos.
  arXiv  Detail & Related papers  (2026-06-01T11:50:16Z)
• Functionality-Oriented LLM Merging on the Fisher--Rao Manifold [14.349284217707575]
  Weight-space merging aims to combine multiple fine-tuned LLMs into a single model without retraining.<n>We derive a practical fixed-point algorithm using a lightweight spherical proxy that preserves norms and generalizes directly to multi-expert merging.
  arXiv  Detail & Related papers  (2026-03-05T09:08:38Z)
• Enhancing Delta Compression in LLMs via SVD-based Quantization Error Minimization [18.037215883638535]
  We introduce DeltaMix, an adaptive mixed-precision delta-compression framework to minimize quantization error in singular value decomposition space.<n>DeltaMix consistently outperforms all baseline methods on tasks such as AIME2024 and GQA.
  arXiv  Detail & Related papers  (2025-06-05T08:17:12Z)
• RGM: A Robust Generalizable Matching Model [49.60975442871967]
  We propose a deep model for sparse and dense matching, termed RGM (Robust Generalist Matching) To narrow the gap between synthetic training samples and real-world scenarios, we build a new, large-scale dataset with sparse correspondence ground truth. We are able to mix up various dense and sparse matching datasets, significantly improving the training diversity.
  arXiv  Detail & Related papers  (2023-10-18T07:30:08Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• On MCMC for variationally sparse Gaussian processes: A pseudo-marginal approach [0.76146285961466]
  Gaussian processes (GPs) are frequently used in machine learning and statistics to construct powerful models. We propose a pseudo-marginal (PM) scheme that offers exact inference as well as computational gains through doubly estimators for the likelihood and large datasets.
  arXiv  Detail & Related papers  (2021-03-04T20:48:29Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Shaping Deep Feature Space towards Gaussian Mixture for Visual Classification [74.48695037007306]
  We propose a Gaussian mixture (GM) loss function for deep neural networks for visual classification. With a classification margin and a likelihood regularization, the GM loss facilitates both high classification performance and accurate modeling of the feature distribution. The proposed model can be implemented easily and efficiently without using extra trainable parameters.
  arXiv  Detail & Related papers  (2020-11-18T03:32:27Z)
• Adversarially-learned Inference via an Ensemble of Discrete Undirected Graphical Models [3.04585143845864]
  We propose an inference-agnostic adversarial training framework which produces an infinitely-large ensemble of graphical models (AGMs) AGMs show significantly better generalization to unseen inference tasks compared to EGMs, as well as deep neural architectures like GibbsNet and VAEAC.
  arXiv  Detail & Related papers  (2020-07-09T19:13:36Z)
• On the benefits of defining vicinal distributions in latent space [20.249590931872568]
  Mixup Training (MT) improves the generalization performance of models by introducing globally linear behavior in between training examples. We propose a new approach - textitVarMixup (Variational Mixup) - to better sample mixup images by using the latent manifold underlying the data. Our empirical studies on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that models trained by performing mixup in the latent manifold learned by VAEs are inherently more robust to various input corruptions/perturbations, are significantly better calibrated, and exhibit more local-linear loss landscapes.
  arXiv  Detail & Related papers  (2020-03-14T06:45:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.