MUSCLE: A Model Update Strategy for Compatible LLM Evolution

• URL: http://arxiv.org/abs/2407.09435v1
• Date: Fri, 12 Jul 2024 17:12:48 GMT
• Title: MUSCLE: A Model Update Strategy for Compatible LLM Evolution
• Authors: Jessica Echterhoff, Fartash Faghri, Raviteja Vemulapalli, Ting-Yao Hu, Chun-Liang Li, Oncel Tuzel, Hadi Pouransari,
• Abstract summary: Large Language Models (LLMs) are frequently updated due to data or architecture changes to improve their performance. Users often build a mental model of the functionality and capabilities of a particular machine learning model they are interacting with. We propose a training strategy to minimize the number of inconsistencies in model updates.
• Score: 29.032461144831053
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are frequently updated due to data or architecture changes to improve their performance. When updating models, developers often focus on increasing overall performance metrics with less emphasis on being compatible with previous model versions. However, users often build a mental model of the functionality and capabilities of a particular machine learning model they are interacting with. They have to adapt their mental model with every update -- a draining task that can lead to user dissatisfaction. In practice, fine-tuned downstream task adapters rely on pretrained LLM base models. When these base models are updated, these user-facing downstream task models experience instance regression or negative flips -- previously correct instances are now predicted incorrectly. This happens even when the downstream task training procedures remain identical. Our work aims to provide seamless model updates to a user in two ways. First, we provide evaluation metrics for a notion of compatibility to prior model versions, specifically for generative tasks but also applicable for discriminative tasks. We observe regression and inconsistencies between different model versions on a diverse set of tasks and model updates. Second, we propose a training strategy to minimize the number of inconsistencies in model updates, involving training of a compatibility model that can enhance task fine-tuned language models. We reduce negative flips -- instances where a prior model version was correct, but a new model incorrect -- by up to 40% from Llama 1 to Llama 2.

Related papers

• GRAM: A Generative Foundation Reward Model for Reward Generalization [48.63394690265176]
  We develop a generative reward model that is first trained via large-scale unsupervised learning and then fine-tuned via supervised learning.<n>This model generalizes well across several tasks, including response ranking, reinforcement learning from human feedback, and task adaptation with fine-tuning.
  arXiv  Detail & Related papers  (2025-06-17T04:34:27Z)
• Intention-Conditioned Flow Occupancy Models [69.79049994662591]
  Large-scale pre-training has fundamentally changed how machine learning research is done today.<n>Applying this same framework to reinforcement learning is appealing because it offers compelling avenues for addressing core challenges in RL.<n>Recent advances in generative AI have provided new tools for modeling highly complex distributions.
  arXiv  Detail & Related papers  (2025-06-10T15:27:46Z)
• Unifying Multimodal Large Language Model Capabilities and Modalities via Model Merging [103.98582374569789]
  Model merging aims to combine multiple expert models into a single model, thereby reducing storage and serving costs.<n>Previous studies have primarily focused on merging visual classification models or Large Language Models (LLMs) for code and math tasks.<n>We introduce the model merging benchmark for MLLMs, which includes multiple tasks such as VQA, Geometry, Chart, OCR, and Grounding, providing both LoRA and full fine-tuning models.
  arXiv  Detail & Related papers  (2025-05-26T12:23:14Z)
• KaLM-Embedding: Superior Training Data Brings A Stronger Embedding Model [27.25688303240741]
  KaLM-Embedding is a general multilingual embedding model that leverages a large quantity of cleaner, more diverse, and domain-specific training data. Our model has been trained with key techniques proven to enhance performance.
  arXiv  Detail & Related papers  (2025-01-02T03:17:51Z)
• MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning [62.78292142632335]
  Class-Incremental Learning (CIL) requires models to continually acquire knowledge of new classes without forgetting old ones. Existing work seeks to utilize lightweight components to adjust the model. We propose MOdel Surgery (MOS) to rescue the model from forgetting previous knowledge.
  arXiv  Detail & Related papers  (2024-12-12T16:57:20Z)
• OLMES: A Standard for Language Model Evaluations [64.85905119836818]
  We propose OLMES, a practical, open standard for reproducible language model evaluations. We identify and review the varying factors in evaluation practices adopted by the community. OLMES supports meaningful comparisons between smaller base models that require the unnatural "cloze" formulation of multiple-choice questions.
  arXiv  Detail & Related papers  (2024-06-12T17:37:09Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Robustness-Congruent Adversarial Training for Secure Machine Learning Model Updates [13.911586916369108]
  We show that misclassifications in machine-learning models can affect robustness to adversarial examples. We propose a technique, named robustness-congruent adversarial training, to address this issue. We show that our algorithm and, more generally, learning with non-regression constraints, provides a theoretically-grounded framework to train consistent estimators.
  arXiv  Detail & Related papers  (2024-02-27T10:37:13Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Revision Transformers: Instructing Language Models to Change their Values [21.645935518842744]
  Current transformer language models (LM) are large-scale models with billions of parameters. We propose the Revision Transformer (RiT) to facilitate easy model updating. The specific combination of a large-scale pre-trained LM that inherently but also diffusely encodes world knowledge with a clear-structured revision engine makes it possible to update the model's knowledge with little effort and the help of user interaction.
  arXiv  Detail & Related papers  (2022-10-19T07:05:06Z)
• Measuring and Reducing Model Update Regression in Structured Prediction for NLP [31.86240946966003]
  backward compatibility requires that the new model does not regress on cases that were correctly handled by its predecessor. This work studies model update regression in structured prediction tasks. We propose a simple and effective method, Backward-Congruent Re-ranking (BCR), by taking into account the characteristics of structured output.
  arXiv  Detail & Related papers  (2022-02-07T07:04:54Z)
• Forward Compatible Training for Representation Learning [53.300192863727226]
  backward compatible training (BCT) modifies training of the new model to make its representations compatible with those of the old model. BCT can significantly hinder the performance of the new model. In this work, we propose a new learning paradigm for representation learning: forward compatible training (FCT)
  arXiv  Detail & Related papers  (2021-12-06T06:18:54Z)
• Mismatched No More: Joint Model-Policy Optimization for Model-Based RL [172.37829823752364]
  We propose a single objective for jointly training the model and the policy, such that updates to either component increases a lower bound on expected return. Our objective is a global lower bound on expected return, and this bound becomes tight under certain assumptions. The resulting algorithm (MnM) is conceptually similar to a GAN.
  arXiv  Detail & Related papers  (2021-10-06T13:43:27Z)
• Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates [68.09049111171862]
  This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. We formulate the regression-free model updates into a constrained optimization problem. We empirically analyze how model ensemble reduces regression.
  arXiv  Detail & Related papers  (2021-05-07T03:33:00Z)
• Self-Updating Models with Error Remediation [0.5156484100374059]
  We propose a framework, Self-Updating Models with Error Remediation (SUMER), in which a deployed model updates itself as new data becomes available. A key component of SUMER is the notion of error remediation as self-labeled data can be susceptible to the propagation of errors. We find that self-updating models (SUMs) generally perform better than models that do not attempt to self-update when presented with additional previously-unseen data.
  arXiv  Detail & Related papers  (2020-05-19T23:09:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.