BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation

• URL: http://arxiv.org/abs/2502.03860v1
• Date: Thu, 06 Feb 2025 08:19:59 GMT
• Title: BOLT: Bootstrap Long Chain-of-Thought in Language Models without Distillation
• Authors: Bo Pang, Hanze Dong, Jiacheng Xu, Silvio Savarese, Yingbo Zhou, Caiming Xiong,
• Abstract summary: o1 from OpenAI has demonstrated remarkable reasoning capabilities. Many teams have attempted to replicate its LongCoT and reasoning capabilities. This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations.
• Score: 88.77999917897702
• License:
• Abstract: Large language models (LLMs), such as o1 from OpenAI, have demonstrated remarkable reasoning capabilities. o1 generates a long chain-of-thought (LongCoT) before answering a question. LongCoT allows LLMs to analyze problems, devise plans, reflect, and backtrack effectively. These actions empower LLM to solve complex problems. After the release of o1, many teams have attempted to replicate its LongCoT and reasoning capabilities. In terms of methods, they primarily rely on knowledge distillation with data from existing models with LongCoT capacities (e.g., OpenAI-o1, Qwen-QwQ, DeepSeek-R1-Preview), leaving significant uncertainties on systematically developing such reasoning abilities. In terms of data domains, these works focus narrowly on math while a few others include coding, limiting their generalizability. This paper introduces a novel approach to enable LLM's LongCoT capacity without distillation from o1-like models or expensive human annotations, where we bootstrap LongCoT (BOLT) from a standard instruct model. BOLT involves three stages: 1) LongCoT data bootstrapping with in-context learning on a standard instruct model; 2) LongCoT supervised finetuning; 3) online training to further refine LongCoT capacities. In BOLT, only a few in-context examples need to be constructed during the bootstrapping stage; in our experiments, we created 10 examples, demonstrating the feasibility of this approach. We use Llama-3.1-70B-Instruct to bootstrap LongCoT and apply our method to various model scales (7B, 8B, 70B). We achieve impressive performance on a variety of benchmarks, Arena-Hard, MT-Bench, WildBench, ZebraLogic, MATH500, which evaluate diverse task-solving and reasoning capabilities.

Related papers

• LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters! [53.84130385074551]
  Large reasoning models (LRMs) tackle complex reasoning problems by following long chain-of-thoughts (Long CoT) We find that a Large Language model (LLM) can effectively learn Long CoT reasoning through data-efficient supervised fine-tuning (SFT) and parameter-efficient low-rank adaptation (LoRA) With just 17k long CoT training samples, the Qwen2.5-32B-Instruct model achieves significant improvements on a wide range of math and coding benchmarks.
  arXiv  Detail & Related papers  (2025-02-11T08:48:48Z)
• RedStar: Does Scaling Long-CoT Data Unlock Better Slow-Reasoning Systems? [40.575978129688586]
  We explore the untapped potential of scaling Long Chain-of-Thought (Long-CoT) data to 1000k samples, pioneering the development of a slow-thinking model, RedStar. Surprisingly, even smaller models show significant performance gains with limited data, revealing the sample efficiency of Long-CoT. RedStar shines across domains: on the MATH-Hard benchmark, RedStar-code-math boosts performance from 66.2% to 81.6%, and on the USA Math Olympiad (AIME) it solves 46.7% of problems using only 21k mixed-code-math datasets.
  arXiv  Detail & Related papers  (2025-01-20T05:44:01Z)
• Bootstrap Your Own Context Length [74.61148597039248]
  We introduce a bootstrapping approach to train long-context language models by exploiting their short-context capabilities only. The proposed data synthesis workflow requires only a short-context language model, a text retriever, and a document collection. We conduct experiments with the open-source Llama-3 family of models and demonstrate that our method can successfully extend the context length to up to 1M tokens.
  arXiv  Detail & Related papers  (2024-12-25T10:08:54Z)
• How to Train Long-Context Language Models (Effectively) [75.5418485597276]
  We study continued training and supervised fine-tuning (SFT) of a language model (LM) to make effective use of long-context information. ProLong-8B, which is from Llama-3 and trained on 40B tokens, demonstrates state-of-the-art long-context performance among similarly sized models at a length of 128K.
  arXiv  Detail & Related papers  (2024-10-03T16:46:52Z)
• LongAlign: A Recipe for Long Context Alignment of Large Language Models [61.85923382850057]
  LongAlign is a recipe of the instruction data, training, and evaluation for long context alignment. We construct a long instruction-following dataset using Self-Instruct. We adopt the packing and sorted strategies to speed up supervised fine-tuning on data with varied length distributions.
  arXiv  Detail & Related papers  (2024-01-31T18:29:39Z)
• LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding [58.20031627237889]
  LongBench is the first bilingual, multi-task benchmark for long context understanding. It comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese)
  arXiv  Detail & Related papers  (2023-08-28T11:53:40Z)
• Giraffe: Adventures in Expanding Context Lengths in LLMs [7.8327063299618]
  We show that linear scaling is the best method for extending context length. We also discover promising extrapolation capabilities in the truncated basis. To support further research in this area, we release three new 13B parameter long-context models.
  arXiv  Detail & Related papers  (2023-08-21T17:30:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.