Slamming: Training a Speech Language Model on One GPU in a Day

• URL: http://arxiv.org/abs/2502.15814v1
• Date: Wed, 19 Feb 2025 17:21:15 GMT
• Title: Slamming: Training a Speech Language Model on One GPU in a Day
• Authors: Gallil Maimon, Avishai Elmakies, Yossi Adi,
• Abstract summary: Slam is a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours.<n>We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost.
• Score: 20.802090523583196
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We introduce Slam, a recipe for training high-quality Speech Language Models (SLMs) on a single academic GPU in 24 hours. We do so through empirical analysis of model initialisation and architecture, synthetic training data, preference optimisation with synthetic data and tweaking all other components. We empirically demonstrate that this training recipe also scales well with more compute getting results on par with leading SLMs in a fraction of the compute cost. We hope these insights will make SLM training and research more accessible. In the context of SLM scaling laws, our results far outperform predicted compute optimal performance, giving an optimistic view to SLM feasibility. See code, data, models, samples at - https://pages.cs.huji.ac.il/adiyoss-lab/slamming .

Related papers

• Scaling Analysis of Interleaved Speech-Text Language Models [22.61336359340435]
  Speech Language Model (SLM) scaling analysis predicts that SLMs require much more compute and data compared to text. Modern SLMs are often initialised from pre-trained TextLMs using speech-text interleaving to allow knowledge transfer. We conduct scaling analysis of interleaved SLMs by training several dozen and analysing the scaling trends. We see that under this setup SLMs scale more efficiently with compute.
  arXiv  Detail & Related papers  (2025-04-03T08:46:56Z)
• Escaping Collapse: The Strength of Weak Data for Large Language Model Training [15.77316232527746]
  We develop a theoretical framework to investigate how much curation is needed in order to ensure that LLM performance continually improves.<n>We describe a training procedure that converges to an optimal LLM even if almost all of the non-synthetic training data is of poor quality.
  arXiv  Detail & Related papers  (2025-02-13T03:20:37Z)
• Scaling Laws for Upcycling Mixture-of-Experts Language Models [17.796361238003403]
  Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters.<n>There are two approaches of mitigating such computational demands: reusing smaller models to train larger ones (upcycling) and training computationally efficient models like mixture-of-experts (MoE)
  arXiv  Detail & Related papers  (2025-02-05T09:11:13Z)
• Sparrow: Data-Efficient Video-LLM with Text-to-Image Augmentation [98.92677830223786]
  This work revisits scaling with synthetic data and focuses on developing video-LLMs from a data-centric perspective. We propose a data augmentation method called Sparrow, which synthesizes video-like samples from pure text instruction data. Our proposed method achieves performance comparable to or even superior to baselines trained with many more samples.
  arXiv  Detail & Related papers  (2024-11-29T18:59:54Z)
• OmniBal: Towards Fast Instruct-tuning for Vision-Language Models via Omniverse Computation Balance [65.48009829137824]
  Large-scale 3D parallel training on vision-language instruct-tuning models leads to an imbalanced computation load across different devices.<n>We rebalanced the computational loads from data, model, and memory perspectives to address this issue.<n>Our method's efficacy and generalizability were further demonstrated across various models and datasets.
  arXiv  Detail & Related papers  (2024-07-30T12:02:58Z)
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models [52.98743860365194]
  We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN) At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents.
  arXiv  Detail & Related papers  (2024-01-02T18:53:13Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Aligning Large Language Models through Synthetic Feedback [43.84431341195111]
  We propose a novel alignment learning framework with synthetic feedback not dependent on extensive human annotations. In human evaluation, our model is preferred to Alpaca and Dolly-v2, 55.0% and 58.5% of the time, respectively.
  arXiv  Detail & Related papers  (2023-05-23T06:41:16Z)
• CodeGen2: Lessons for Training LLMs on Programming and Natural Languages [116.74407069443895]
  We unify encoder and decoder-based models into a single prefix-LM. For learning methods, we explore the claim of a "free lunch" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored.
  arXiv  Detail & Related papers  (2023-05-03T17:55:25Z)
• Cramming: Training a Language Model on a Single GPU in One Day [64.18297923419627]
  Recent trends in language modeling have focused on increasing performance through scaling. We investigate the downstream performance achievable with a transformer-based language model trained completely from scratch with masked language modeling for a single day on a single consumer GPU. We provide evidence that even in this constrained setting, performance closely follows scaling laws observed in large-compute settings.
  arXiv  Detail & Related papers  (2022-12-28T18:59:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.