FeatCal: Feature Calibration for Post-Merging Models

• URL: http://arxiv.org/abs/2605.13030v1
• Date: Wed, 13 May 2026 05:35:03 GMT
• Title: FeatCal: Feature Calibration for Post-Merging Models
• Abstract summary: Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models.<n>We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input.<n>This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order.
• Score: 37.524690106426284
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model merging combines task experts into one model and avoids joint training, retraining, or deploying many expert models, but the merged model often still underperforms task experts. We study this performance gap through feature drift, the difference between features produced by the merged model and by the expert on the same input. Our theory decomposes this drift into upstream propagation and local mismatch, tracks how it propagates and combines through later layers in forward order, and links final feature drift to output drift. This view motivates FeatCal, which uses a small calibration set to calibrate the merged model weights layer by layer in forward order, reducing feature drift while staying close to merged weights and preserving the benefits of model merging. FeatCal uses an efficient closed-form solution to update model weights, with no gradient descent, iterative optimization, or extra modules. On the main CLIP and GLUE benchmarks, FeatCal beats Surgery and ProbSurgery, the closest post-merging calibration baselines: 85.5% vs. 77.0%/78.8% on CLIP-ViT-B/32 Task Arithmetic (TA) and 85.2% vs. 83.7%/82.2% on FLAN-T5-base GLUE. On CLIP-ViT-B/32, 8 examples per task reach 82.9%, and 256 examples per task take 53 seconds, about 4x faster than both baselines, showing better sample efficiency and lower calibration cost.

Related papers

• E-PMQ: Expert-Guided Post-Merge Quantization with Merged-Weight Anchoring [25.088494564026423]
  We show that Post-Merge Quantization (PMQ) enables effective post-merge quantization and low-bit deployment.<n>We propose E-PMQ, an expert-guided PMQ framework that uses source expert weights to provide expert- guided output targets.<n>These results demonstrate that E-PMQ enables effective post-merge quantization and low-bit deployment.
  arXiv  Detail & Related papers  (2026-05-16T08:44:36Z)
• MAGIC: Achieving Superior Model Merging via Magnitude Calibration [46.30162864565835]
  Model merging aims to merge the capabilities of specialised models into a unified model.<n>We identify that features consist of two critical components: direction and magnitude.<n>We propose MAGnItude (MAGIC), a plug-and-play framework that rectifies layer-wise magnitudes in feature and weight spaces.
  arXiv  Detail & Related papers  (2025-12-22T12:13:17Z)
• The Few Govern the Many:Unveiling Few-Layer Dominance for Time Series Models [14.479996384788445]
  We observe a puzzling phenomenon that larger models do emphNOT achieve better performance.<n>We propose a practical method to automatically identify and retain only these dominant layers.
  arXiv  Detail & Related papers  (2025-11-10T15:53:29Z)
• Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correction [95.91743732150233]
  Goedel-Prover-V2, a series of open-source language models, set a new state-of-the-art in automated theorem proving.<n>We generate synthetic tasks of increasing difficulty to train the model to master increasingly complex theorems.<n>Goedel-Prover-V2-32B achieves 88.1% on MiniF2F at pass@32 in standard mode and 90.4% in self-correction mode.
  arXiv  Detail & Related papers  (2025-08-05T16:28:22Z)
• Multi-Level Collaboration in Model Merging [56.31088116526825]
  This paper explores the intrinsic connections between model merging and model ensembling.<n>We find that even when previous restrictions are not met, there is still a way for model merging to attain a near-identical and superior performance similar to that of ensembling.
  arXiv  Detail & Related papers  (2025-03-03T07:45:04Z)
• Entropy Adaptive Decoding: Dynamic Model Switching for Efficient Inference [0.0]
  We present Entropy Adaptive Decoding (EAD), a novel approach for efficient language model inference.<n>EAD switches between different-sized models based on prediction uncertainty.<n>We show remarkable efficiency gains across different model families.
  arXiv  Detail & Related papers  (2025-02-05T22:15:21Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Aligner: Efficient Alignment by Learning to Correct [10.056049435141645]
  We introduce Aligner, a model-agnostic, plug-and-play module that learns the correctional residuals between preferred and dispreferred answers. It can be applied to various open-source and API-based models with only one-off training, making it suitable for rapid iteration. Our experiments demonstrate performance improvements by deploying the same Aligner model across 11 different language models.
  arXiv  Detail & Related papers  (2024-02-04T09:24:51Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Training with Quantization Noise for Extreme Model Compression [57.51832088938618]
  We tackle the problem of producing compact models, maximizing their accuracy for a given model size. A standard solution is to train networks with Quantization Aware Training, where the weights are quantized during training and the gradients approximated with the Straight-Through Estimator. In this paper, we extend this approach to work beyond int8 fixed-point quantization with extreme compression methods.
  arXiv  Detail & Related papers  (2020-04-15T20:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.