FLM-101B: An Open LLM and How to Train It with $100K Budget

• URL: http://arxiv.org/abs/2309.03852v3
• Date: Tue, 14 Jan 2025 06:40:36 GMT
• Title: FLM-101B: An Open LLM and How to Train It with $100K Budget
• Authors: Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun, Yequan Wang,
• Abstract summary: We show that FLM-101B, trained with our growth strategy under a budget of $100K, reaches 80% of the baselines' performances with only 10% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI.
• Score: 63.244403881531035
• License:
• Abstract: Large language models (LLMs) are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of \$100K, reaches 80\% of the baselines' performances with only 10\% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B.

Related papers

• CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation [17.807249890437767]
  We introduce CoLA and its memory-efficient implementation, CoLA-M. We leverage the low-rank structure observed widely in model activations to reduce model size, boost model capacity and training efficiency. Experiments on LLaMA models with 60 million to 7 billion parameters show that CoLA reduces the computing cost by $bf 2pmbtimes$ and improves training throughput by $bf 1.86pmbtimes$ while maintaining full-rank level performance.
  arXiv  Detail & Related papers  (2025-02-16T01:05:16Z)
• Control LLM: Controlled Evolution for Intelligence Retention in LLM [4.67235851066221]
  We propose textbfControl LLM, a novel approach that leverages parallel pre-trained and expanded transformer blocks. Experiments demonstrate the effectiveness of Control LLM in both Continuous Pre-training (CPT) and Continuous Supervised Fine-Tuning (CSFT) It surpasses existing methods and achieves SOTA among open-source models tuned from the same base model, using substantially less data and compute.
  arXiv  Detail & Related papers  (2025-01-19T08:06:06Z)
• Sparsity-Accelerated Training for Large Language Models [20.86225596276327]
  Large language models (LLMs) have demonstrated proficiency across various natural language processing (NLP) tasks. LLMs often require additional training, such as continual pre-training and supervised fine-tuning. This paper proposes leveraging emphsparsity in pre-trained LLMs to expedite this training process.
  arXiv  Detail & Related papers  (2024-06-03T14:56:09Z)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems [59.40480894948944]
  Large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. We prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning.
  arXiv  Detail & Related papers  (2024-05-30T09:42:54Z)
• MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT [87.4910758026772]
  "Bigger the better" has been the predominant trend in recent Large Language Models (LLMs) development. This paper explores the "less is more" paradigm by addressing the challenge of designing accurate yet efficient Small Language Models (SLMs) for resource constrained devices.
  arXiv  Detail & Related papers  (2024-02-26T18:59:03Z)
• Optimizing Distributed Training on Frontier for Large Language Models [7.251642875697334]
  Training large language models (LLMs) with billions of parameters poses significant challenges and requires considerable computational resources. This research explores efficient distributed training strategies to extract this computation from Frontier, the world's first exascale supercomputer.
  arXiv  Detail & Related papers  (2023-12-20T02:03:15Z)
• Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own [59.11934130045106]
  We propose Reinforcement Learning with Foundation Priors (RLFP) to utilize guidance and feedback from policy, value, and success-reward foundation models. Within this framework, we introduce the Foundation-guided Actor-Critic (FAC) algorithm, which enables embodied agents to explore more efficiently with automatic reward functions. Our method achieves remarkable performances in various manipulation tasks on both real robots and in simulation.
  arXiv  Detail & Related papers  (2023-10-04T07:56:42Z)
• GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length [65.24730341801468]
  This paper introduces a novel, simple, and effective method named growlength'' to accelerate the pretraining process of Large Language Models. Our method progressively increases the training length throughout the pretraining phase, thereby mitigating computational costs and enhancing efficiency.
  arXiv  Detail & Related papers  (2023-10-01T05:25:24Z)
• Knowledge Inheritance for Pre-trained Language Models [57.51305807391381]
  We introduce a novel pre-training framework named "knowledge inheritance" (KI) KI combines both self-learning and teacher-guided learning to efficiently train larger PLMs. We show that KI can well support lifelong learning and knowledge transfer.
  arXiv  Detail & Related papers  (2021-05-28T14:43:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.