O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?

• URL: http://arxiv.org/abs/2411.16489v1
• Date: Mon, 25 Nov 2024 15:31:27 GMT
• Title: O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
• Authors: Zhen Huang, Haoyang Zou, Xuefeng Li, Yixiu Liu, Yuxiang Zheng, Ethan Chern, Shijie Xia, Yiwei Qin, Weizhe Yuan, Pengfei Liu,
• Abstract summary: This paper presents a critical examination of current approaches to replicating OpenAI's O1 model capabilities. We show how simple distillation from O1's API, combined with supervised fine-tuning, can achieve superior performance on complex mathematical reasoning tasks.
• Score: 30.87379989964516
• License:
• Abstract: This paper presents a critical examination of current approaches to replicating OpenAI's O1 model capabilities, with particular focus on the widespread but often undisclosed use of knowledge distillation techniques. While our previous work explored the fundamental technical path to O1 replication, this study reveals how simple distillation from O1's API, combined with supervised fine-tuning, can achieve superior performance on complex mathematical reasoning tasks. Through extensive experiments, we show that a base model fine-tuned on simply tens of thousands of samples O1-distilled long-thought chains outperforms O1-preview on the American Invitational Mathematics Examination (AIME) with minimal technical complexity. Moreover, our investigation extends beyond mathematical reasoning to explore the generalization capabilities of O1-distilled models across diverse tasks: hallucination, safety and open-domain QA. Notably, despite training only on mathematical problem-solving data, our models demonstrated strong generalization to open-ended QA tasks and became significantly less susceptible to sycophancy after fine-tuning. We deliberately make this finding public to promote transparency in AI research and to challenge the current trend of obscured technical claims in the field. Our work includes: (1) A detailed technical exposition of the distillation process and its effectiveness, (2) A comprehensive benchmark framework for evaluating and categorizing O1 replication attempts based on their technical transparency and reproducibility, (3) A critical discussion of the limitations and potential risks of over-relying on distillation approaches, our analysis culminates in a crucial bitter lesson: while the pursuit of more capable AI systems is important, the development of researchers grounded in first-principles thinking is paramount.

Related papers

• O1 Embedder: Let Retrievers Think Before Action [28.583031173137428]
  We propose O1 Embedder, which generates useful thoughts for the input query before making retrieval for the target documents. Our approach is evaluated by comprehensive experiments, where substantial improvements are achieved across 12 popular datasets. These results highlight O1 Embedder's remarkable accuracy and generalizability, paving the way for the development of next-generation IR foundation models.
  arXiv  Detail & Related papers  (2025-02-11T13:48:10Z)
• Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs [76.43407125275202]
  o1-like models can emulate human-like long-time thinking during inference. This paper presents the first comprehensive study on the prevalent issue of overthinking in these models. We propose strategies to mitigate overthinking, streamlining reasoning processes without compromising accuracy.
  arXiv  Detail & Related papers  (2024-12-30T18:55:12Z)
• Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspective [77.94874338927492]
  OpenAI has claimed that the main techinique behinds o1 is the reinforcement learning. This paper analyzes the roadmap to achieving o1 from the perspective of reinforcement learning.
  arXiv  Detail & Related papers  (2024-12-18T18:24:47Z)
• Imitate, Explore, and Self-Improve: A Reproduction Report on Slow-thinking Reasoning Systems [92.89673285398521]
  o1-like reasoning systems have demonstrated remarkable capabilities in solving complex reasoning tasks. We introduce an imitate, explore, and self-improve'' framework to train the reasoning model. Our approach achieves competitive performance compared to industry-level reasoning systems.
  arXiv  Detail & Related papers  (2024-12-12T16:20:36Z)
• A Comparative Study on Reasoning Patterns of OpenAI's o1 Model [69.08287909042421]
  We show that OpenAI's o1 model has achieved the best performance on most datasets. We also provide a detailed analysis on several reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-10-17T15:09:03Z)
• O1 Replication Journey: A Strategic Progress Report -- Part 1 [52.062216849476776]
  This paper introduces a pioneering approach to artificial intelligence research, embodied in our O1 Replication Journey. Our methodology addresses critical challenges in modern AI research, including the insularity of prolonged team-based projects. We propose the journey learning paradigm, which encourages models to learn not just shortcuts, but the complete exploration process.
  arXiv  Detail & Related papers  (2024-10-08T15:13:01Z)
• Evaluation of OpenAI o1: Opportunities and Challenges of AGI [112.0812059747033]
  o1-preview demonstrated remarkable capabilities, often achieving human-level or superior performance. The model excelled in tasks requiring intricate reasoning and knowledge integration across various fields. Overall results indicate significant progress towards artificial general intelligence.
  arXiv  Detail & Related papers  (2024-09-27T06:57:00Z)
• Simple Unsupervised Similarity-Based Aspect Extraction [0.9558392439655015]
  We propose a simple approach called SUAEx for aspect extraction. SUAEx is unsupervised and relies solely on the similarity of word embeddings. Experimental results on datasets from three different domains have shown that SUAEx achieves results that can outperform the state-of-the-art attention-based approach at a fraction of the time.
  arXiv  Detail & Related papers  (2020-08-25T04:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.