Generalizing the Geometry of Model Merging Through Frechet Averages

• URL: http://arxiv.org/abs/2604.27155v2
• Date: Thu, 07 May 2026 12:23:33 GMT
• Title: Generalizing the Geometry of Model Merging Through Frechet Averages
• Abstract summary: We show that not only the geometry, but also the averaging procedure itself, must be symmetry-invariant to achieve symmetry-aware merges.<n>We propose a general solution: merging as Fréchet averaging, selecting parameters that minimize a sum of geodesic distances on an appropriate manifold.
• Score: 13.893485499000219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model merging aims to combine multiple models into one without additional training. Naïve parameter-space averaging can be fragile under architectural symmetries, as their geometry does not take them into account. In this work we show that not only the geometry, but also the averaging procedure itself, must be symmetry-invariant to achieve symmetry-aware merges. Consequently, we propose a general solution: merging as Fréchet averaging, i.e., selecting parameters that minimize a sum of geodesic distances on an appropriate manifold. In this view, the key design choice is the overall geometry, i.e., the choice of metric, manifold, and distance approximation, that determines what it means for two models to be "close". We show that Fréchet averaging, combined with simplifying assumptions, contains Fisher merging. Building on this, we examine the particular case of low-rank adapters (LoRA), whose symmetries induce a distinct geometry: that of a quotient manifold. We outline the limitations of current LoRA merging methods, propose a practical algorithm for this setting, and show how they compare with other commonly used approaches.

Related papers

• Riemannian Optimization for Hadamard Products of Low-Rank Matrices [8.444687277893827]
  An elementwise Hadamard product of two low-rank matrices provides a parameter-efficient model for data with multiplicative structure, but its modeling is challenging due to the presence of additional symmetries.<n>In order to leverage the geometry of the space, we formulate the learning of such matrices as optimization on a Riemannian quotient manifold.<n>We develop a gradient-free Gauss--Newton descent algorithm that uses a tuning-free Gauss--Newton step size and scales linearly in the number of observed entries per iteration.
  arXiv  Detail & Related papers  (2026-05-31T13:10:23Z)
• Midpoint Generative Models [52.59974039570536]
  We introduce Midpoint Generative Models (MGM), a principled framework for training one-step generative models.<n>MGM offers an effective and theoretically grounded approach to generative modeling, achieving competitive performance against existing one-step generative modeling methods.
  arXiv  Detail & Related papers  (2026-05-28T13:31:53Z)
• Model Merging on Loss Landscape: A Geometry Perspective [2.2624434825206543]
  EpiMer is a framework that casts model merging as solving the Fréchet mean on a Riemannian manifold.<n>Our theoretical analysis decomposes the merging error bound into the subspace Fréchet variance and the residual energy.<n>Merging fine-tuned CLIP-ViT models on eight image classification tasks, Epistemic Merging strictly outperforms the baselines on all three CLIP-ViT backbones at matched rank.
  arXiv  Detail & Related papers  (2026-05-26T08:33:30Z)
• Riemannian Consistency Model [57.933800575074535]
  We propose the Riemannian Consistency Model (RCM), which, for the first time, enables few-step consistency modeling.<n>We derive the closed-form solutions for both discrete- and continuous-time training objectives for RCM.<n>We provide a unique kinematics perspective for interpreting the RCM objective, offering new theoretical angles.
  arXiv  Detail & Related papers  (2025-10-01T14:57:25Z)
• Bayesian Circular Regression with von Mises Quasi-Processes [57.88921637944379]
  In this work we explore a family of expressive and interpretable distributions over circle-valued random functions.<n>For posterior inference, we introduce a new Stratonovich-like augmentation that lends itself to fast Gibbs sampling.<n>We present experiments applying this model to the prediction of wind directions and the percentage of the running gait cycle as a function of joint angles.
  arXiv  Detail & Related papers  (2024-06-19T01:57:21Z)
• Scaling Riemannian Diffusion Models [68.52820280448991]
  We show that our method enables us to scale to high dimensional tasks on nontrivial manifold. We model QCD densities on $SU(n)$ lattices and contrastively learned embeddings on high dimensional hyperspheres.
  arXiv  Detail & Related papers  (2023-10-30T21:27:53Z)
• The Fisher-Rao geometry of CES distributions [50.50897590847961]
  The Fisher-Rao information geometry allows for leveraging tools from differential geometry. We will present some practical uses of these geometric tools in the framework of elliptical distributions.
  arXiv  Detail & Related papers  (2023-10-02T09:23:32Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• Regularizing Towards Soft Equivariance Under Mixed Symmetries [23.603875905608565]
  We present a regularizer-based method for building a model for a dataset with mixed approximate symmetries. We show that our method achieves better accuracy than prior approaches while discovering the approximate symmetry levels correctly.
  arXiv  Detail & Related papers  (2023-06-01T05:33:41Z)
• Flow Matching on General Geometries [43.252817099263744]
  We propose a simple yet powerful framework for training continuous normalizing flows on manifold geometries. We show that it is simulation-free on simple geometries, does not require divergence, and computes its target vector field in closed-form. Our method achieves state-of-the-art performance on many real-world non-Euclidean datasets.
  arXiv  Detail & Related papers  (2023-02-07T18:21:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.