Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance

• URL: http://arxiv.org/abs/2511.13254v1
• Date: Mon, 17 Nov 2025 11:13:34 GMT
• Title: Souper-Model: How Simple Arithmetic Unlocks State-of-the-Art LLM Performance
• Authors: Shalini Maiti, Amar Budhiraja, Bhavul Gauri, Gaurav Chaurasia, Anton Protopopov, Alexis Audran-Reiss, Michael Slater, Despoina Magka, Tatiana Shavrina, Roberta Raileanu, Yoram Bachrach,
• Abstract summary: We introduce Soup Of Category Experts (SoCE), a principled approach for model souping.<n>SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging.<n>We demonstrate that the proposed method improves performance and robustness across multiple domains.
• Score: 15.244354622288226
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse domains, but their training remains resource- and time-intensive, requiring massive compute power and careful orchestration of training procedures. Model souping-the practice of averaging weights from multiple models of the same architecture-has emerged as a promising pre- and post-training technique that can enhance performance without expensive retraining. In this paper, we introduce Soup Of Category Experts (SoCE), a principled approach for model souping that utilizes benchmark composition to identify optimal model candidates and applies non-uniform weighted averaging to maximize performance. Contrary to previous uniform-averaging approaches, our method leverages the observation that benchmark categories often exhibit low inter-correlations in model performance. SoCE identifies "expert" models for each weakly-correlated category cluster and combines them using optimized weighted averaging rather than uniform weights. We demonstrate that the proposed method improves performance and robustness across multiple domains, including multilingual capabilities, tool calling, and math and achieves state-of-the-art results on the Berkeley Function Calling Leaderboard.

Related papers

• DiRL: An Efficient Post-Training Framework for Diffusion Language Models [54.405206032785706]
  Diffusion Language Models (dLLMs) have emerged as promising alternatives to Auto-Regressive (AR) models.<n>Existing methods suffer from computational inefficiency and objective mismatches between training and inference.<n>We introduce DiRL, an efficient post-training framework that tightly integrates FlexAttention-accelerated blockwise training with LMDeploy-optimized inference.
  arXiv  Detail & Related papers  (2025-12-23T08:33:19Z)
• Parameter Averaging in Link Prediction [7.914373583039709]
  We introduce model merging, specifically weighted averaging, in knowledge graph embedding (KGE) models.<n>Running average of model parameters from a training epoch onward is maintained and used for predictions.<n>We evaluate these two different weighted averaging approaches on link prediction tasks, comparing the state-of-the-art benchmark ensemble approach.
  arXiv  Detail & Related papers  (2025-10-29T10:32:39Z)
• Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization [60.309915093470416]
  Matryoshka MoE (M-MoE) is a training framework that instills a coarse-to-fine structure directly into the expert ensemble.<n>Our work paves the way for more practical and adaptable deployments of large-scale MoE models.
  arXiv  Detail & Related papers  (2025-09-30T16:56:44Z)
• SPaRFT: Self-Paced Reinforcement Fine-Tuning for Large Language Models [51.74498855100541]
  Large language models (LLMs) have shown strong reasoning capabilities when fine-tuned with reinforcement learning (RL)<n>We propose textbfSPaRFT, a self-paced learning framework that enables efficient learning based on the capability of the model being trained.
  arXiv  Detail & Related papers  (2025-08-07T03:50:48Z)
• Combining Pre-Trained Models for Enhanced Feature Representation in Reinforcement Learning [16.04558746520946]
  Reinforcement Learning (RL) focuses on maximizing the cumulative reward obtained via agent's interaction with the environment.<n>We propose Weight Sharing Attention (WSA), a new architecture to combine embeddings of multiple pre-trained models to shape an enriched state representation.
  arXiv  Detail & Related papers  (2025-07-09T18:13:52Z)
• Relative Overfitting and Accept-Reject Framework [5.465098504510676]
  We propose an ensemble framework that governs how models are segmented to ensure performance improvement.<n>We detail the patterns of this framework within the domain of NLP and briefly describe its to other fields, such as computer vision (CV) and AI for science.
  arXiv  Detail & Related papers  (2025-05-12T17:36:14Z)
• DeepInsert: Early Layer Bypass for Efficient and Performant Multimodal Understanding [26.39397960987363]
  We propose a simple modification to pretrained transformer models.<n>Instead of concatenation with the language prompt at the start, we insert multimodal tokens directly into the middle.<n>Our results indicate that our method reduces computational costs during both training and inference.
  arXiv  Detail & Related papers  (2025-04-27T18:56:26Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Instruction-Following Pruning for Large Language Models [58.329978053711024]
  We move beyond the traditional static pruning approach of determining a fixed pruning mask for a model.<n>In our method, the pruning mask is input-dependent and adapts dynamically based on the information described in a user instruction.<n>Our approach, termed "instruction-following pruning", introduces a sparse mask predictor that takes the user instruction as input and dynamically selects the most relevant model parameters for the given task.
  arXiv  Detail & Related papers  (2025-01-03T20:19:14Z)
• Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training [32.154166415680066]
  Methods like distillation, compression or quantization help leverage the highly performant large models to induce smaller performant ones. This paper explores the hypothesis that a single training run can simultaneously train a larger model for performance and derive a smaller model for deployment.
  arXiv  Detail & Related papers  (2024-02-07T17:07:41Z)
• One-stop Training of Multiple Capacity Models [74.87789190840527]
  We propose a novel one-stop training framework to jointly train high-capacity and low-capactiy models. Unlike knowledge distillation, where multiple capacity models are trained from scratch separately, our approach integrates supervisions from different capacity models simultaneously.
  arXiv  Detail & Related papers  (2023-05-23T13:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.