Rho-1: Not All Tokens Are What You Need

• URL: http://arxiv.org/abs/2404.07965v2
• Date: Thu, 23 May 2024 06:57:07 GMT
• Title: Rho-1: Not All Tokens Are What You Need
• Authors: Zhenghao Lin, Zhibin Gou, Yeyun Gong, Xiao Liu, Yelong Shen, Ruochen Xu, Chen Lin, Yujiu Yang, Jian Jiao, Nan Duan, Weizhu Chen,
• Abstract summary: Previous language model pre-training methods uniformly applied a next-token prediction loss to all training tokens. Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM) Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks.
• Score: 132.31428897792114
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Previous language model pre-training methods have uniformly applied a next-token prediction loss to all training tokens. Challenging this norm, we posit that ''Not all tokens in a corpus are equally important for language model training''. Our initial analysis examines token-level training dynamics of language model, revealing distinct loss patterns for different tokens. Leveraging these insights, we introduce a new language model called Rho-1. Unlike traditional LMs that learn to predict every next token in a corpus, Rho-1 employs Selective Language Modeling (SLM), which selectively trains on useful tokens that aligned with the desired distribution. This approach involves scoring pretraining tokens using a reference model, and then training the language model with a focused loss on tokens with higher scores. When continual pretraining on 15B OpenWebMath corpus, Rho-1 yields an absolute improvement in few-shot accuracy of up to 30% in 9 math tasks. After fine-tuning, Rho-1-1B and 7B achieved state-of-the-art results of 40.6% and 51.8% on MATH dataset, respectively - matching DeepSeekMath with only 3% of the pretraining tokens. Furthermore, when pretraining on 80B general tokens, Rho-1 achieves 6.8% average enhancement across 15 diverse tasks, increasing both efficiency and performance of the language model pre-training.

Related papers

• Patch-Level Training for Large Language Models [69.67438563485887]
  This paper introduces patch-level training for Large Language Models (LLMs) During patch-level training, we feed the language model shorter sequences of patches and train it to predict the next patch. Following this, the model continues token-level training on the remaining training data to align with the inference mode.
  arXiv  Detail & Related papers  (2024-07-17T15:48:39Z)
• Better & Faster Large Language Models via Multi-token Prediction [29.067271500844928]
  Large language models such as GPT and Llama are trained with a next-token prediction loss. We suggest that training language models to predict multiple future tokens at once results in higher sample efficiency.
  arXiv  Detail & Related papers  (2024-04-30T17:33:57Z)
• MiLe Loss: a New Loss for Mitigating the Bias of Learning Difficulties in Generative Language Models [40.992566245706996]
  We propose a MiLe Loss function for mitigating the bias of learning difficulties with tokens. We train generative language models at different scales of 468M, 1.2B, and 6.7B parameters. Experiments reveal that models incorporating the proposed MiLe Loss can gain consistent performance improvement on downstream benchmarks.
  arXiv  Detail & Related papers  (2023-10-30T13:33:21Z)
• Understanding the Role of Input Token Characters in Language Models: How Does Information Loss Affect Performance? [45.53600782873268]
  We study how information loss in input token characters affects the performance of pre-training language models. Surprisingly, we find that pre-training even under extreme settings, i.e. using only one character of each token, the performance retention in standard NLU benchmarks and probing tasks is high. For instance, a model pre-trained only on single first characters from tokens achieves performance retention of approximately $90$% and $77$% of the full-token model in SuperGLUE and GLUE tasks, respectively.
  arXiv  Detail & Related papers  (2023-10-26T09:47:50Z)
• Self-Evolution Learning for Discriminative Language Model Pretraining [103.57103957631067]
  Self-Evolution learning (SE) is a simple and effective token masking and learning method. SE focuses on learning the informative yet under-explored tokens and adaptively regularizes the training by introducing a novel Token-specific Label Smoothing approach.
  arXiv  Detail & Related papers  (2023-05-24T16:00:54Z)
• FCM: Forgetful Causal Masking Makes Causal Language Models Better Zero-Shot Learners [139.6321017962092]
  We propose a simple technique that significantly boosts the performance of large language models without adding computational cost. Our key observation is that, by performing the next token prediction task with randomly selected past tokens masked out, we can improve the quality of the learned representations. Experimental results show that our method also improves PaLM's zero and few-shot performance on a diverse suite of tasks.
  arXiv  Detail & Related papers  (2022-10-24T17:46:57Z)
• ELECTRA is a Zero-Shot Learner, Too [14.315501760755609]
  "Pre-train, prompt, and predict" has achieved remarkable achievements compared with the "pre-train, fine-tune" paradigm. In this paper, we propose a novel replaced token detection (RTD)-based prompt learning method. Experimental results show that ELECTRA model based onRTD-prompt learning achieves surprisingly state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2022-07-17T11:20:58Z)
• Training Compute-Optimal Large Language Models [54.00424650998489]
  We train language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens. For compute-optimal training, the model size and the number of training tokens should be scaled equally. chinchilla uniformly and significantly outperforms Gopher (280B), GPT-3 (175B), Jurassic-1 (178B), and Megatron-Turing NLG (530B)
  arXiv  Detail & Related papers  (2022-03-29T13:38:03Z)
• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators [108.3381301768299]
  Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. We propose a more sample-efficient pre-training task called replaced token detection.
  arXiv  Detail & Related papers  (2020-03-23T21:17:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.