Mashup Learning: Faster Finetuning by Remixing Past Checkpoints

• URL: http://arxiv.org/abs/2603.10156v1
• Date: Tue, 10 Mar 2026 18:46:26 GMT
• Title: Mashup Learning: Faster Finetuning by Remixing Past Checkpoints
• Abstract summary: Mashup Learning is a simple method to leverage the outputs of prior training runs to enhance model adaptation to new tasks.<n>It consistently improves average downstream accuracy by 0.5-5 percentage points over training from scratch.<n>It also accelerates convergence, requiring 41-46% fewer training steps and up to 37% less total wall-clock time to match from-scratch accuracy.
• Score: 11.36221733394798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Finetuning on domain-specific data is a well-established method for enhancing LLM performance on downstream tasks. Training on each dataset produces a new set of model weights, resulting in a multitude of checkpoints saved in-house or on open-source platforms. However, these training artifacts are rarely reused for subsequent experiments despite containing improved model abilities for potentially similar tasks. In this paper, we propose Mashup Learning, a simple method to leverage the outputs of prior training runs to enhance model adaptation to new tasks. Our procedure identifies the most relevant historical checkpoints for a target dataset, aggregates them with model merging, and uses the result as an improved initialization for training. Across 8 standard LLM benchmarks, four models, and two collections of source checkpoints, Mashup Learning consistently improves average downstream accuracy by 0.5-5 percentage points over training from scratch. It also accelerates convergence, requiring 41-46% fewer training steps and up to 37% less total wall-clock time to match from-scratch accuracy, including all selection and merging overhead.

Related papers

• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference.<n>Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• FREE: Faster and Better Data-Free Meta-Learning [77.90126669914324]
  Data-Free Meta-Learning (DFML) aims to extract knowledge from a collection of pre-trained models without requiring the original data.<n>We introduce the Faster and Better Data-Free Meta-Learning framework, which contains: (i) a meta-generator for rapidly recovering training tasks from pre-trained models; and (ii) a meta-learner for generalizing to new unseen tasks.
  arXiv  Detail & Related papers  (2024-05-02T03:43:19Z)
• Bad Students Make Great Teachers: Active Learning Accelerates Large-Scale Visual Understanding [9.112203072394648]
  Power-law scaling indicates that large-scale training with uniform sampling is prohibitively slow. Active learning methods aim to increase data efficiency by prioritizing learning on the most relevant examples.
  arXiv  Detail & Related papers  (2023-12-08T19:26:13Z)
• DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detection [72.25697820290502]
  This work introduces a straightforward and efficient strategy to identify potential novel classes through zero-shot classification. We refer to this approach as the self-training strategy, which enhances recall and accuracy for novel classes without requiring extra annotations, datasets, and re-training. Empirical evaluations on three datasets, including LVIS, V3Det, and COCO, demonstrate significant improvements over the baseline performance.
  arXiv  Detail & Related papers  (2023-10-02T17:52:24Z)
• D4: Improving LLM Pretraining via Document De-Duplication and Diversification [38.84592304799403]
  We show that careful data selection via pre-trained model embeddings can speed up training. We also show that repeating data intelligently consistently outperforms baseline training.
  arXiv  Detail & Related papers  (2023-08-23T17:58:14Z)
• NLU on Data Diets: Dynamic Data Subset Selection for NLP Classification Tasks [0.0]
  Finetuning large language models inflates the costs of NLU applications. Recent works in computer vision use data pruning to reduce training time. We propose a curriculum which periodically scores and discards unimportant examples during finetuning.
  arXiv  Detail & Related papers  (2023-06-05T19:30:41Z)
• Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learning [30.65315081964461]
  We study few-shot learning with pretrained language models (PLMs) from a different perspective. We first tune an autoregressive PLM on the few-shot samples and then use it as a generator to synthesize a large amount of novel training samples. Our approach FewGen achieves an overall better result across seven classification tasks of the GLUE benchmark than existing few-shot learning methods.
  arXiv  Detail & Related papers  (2022-11-06T06:46:47Z)
• Jigsaw Clustering for Unsupervised Visual Representation Learning [68.09280490213399]
  We propose a new jigsaw clustering pretext task in this paper. Our method makes use of information from both intra- and inter-images. It is even comparable to the contrastive learning methods when only half of training batches are used.
  arXiv  Detail & Related papers  (2021-04-01T08:09:26Z)
• Meta-Generating Deep Attentive Metric for Few-shot Classification [53.07108067253006]
  We present a novel deep metric meta-generation method to generate a specific metric for a new few-shot learning task. In this study, we structure the metric using a three-layer deep attentive network that is flexible enough to produce a discriminative metric for each task. We gain surprisingly obvious performance improvement over state-of-the-art competitors, especially in the challenging cases.
  arXiv  Detail & Related papers  (2020-12-03T02:07:43Z)
• Cross-Domain Few-Shot Learning with Meta Fine-Tuning [8.062394790518297]
  We tackle the new Cross-Domain Few-Shot Learning benchmark proposed by the CVPR 2020 Challenge. We build upon state-of-the-art methods in domain adaptation and few-shot learning to create a system that can be trained to perform both tasks.
  arXiv  Detail & Related papers  (2020-05-21T09:55:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.