Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models

• URL: http://arxiv.org/abs/2412.02674v1
• Date: Tue, 03 Dec 2024 18:47:26 GMT
• Title: Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
• Authors: Yuda Song, Hanlin Zhang, Carson Eisenach, Sham Kakade, Dean Foster, Udaya Ghai,
• Abstract summary: Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance.
• Score: 10.449015816015566
• License:
• Abstract: Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries.

Related papers

• ReVISE: Learning to Refine at Test-Time via Intrinsic Self-Verification [53.80183105328448]
  Refine via Intrinsic Self-Verification (ReVISE) is an efficient framework that enables LLMs to self-correct their outputs through self-verification. Our experiments on various reasoning tasks demonstrate that ReVISE achieves efficient self-correction and significantly improves reasoning performance.
  arXiv  Detail & Related papers  (2025-02-20T13:50:02Z)
• S$^2$R: Teaching LLMs to Self-verify and Self-correct via Reinforcement Learning [51.84977135926156]
  We introduce S$2$R, an efficient framework that enhances LLM reasoning by teaching models to self-verify and self-correct during inference. Our results demonstrate that Qwen2.5-math-7B achieves an accuracy improvement from 51.0% to 81.6%, outperforming models trained on an equivalent amount of long-CoT distilled data.
  arXiv  Detail & Related papers  (2025-02-18T13:40:22Z)
• Iterative Deepening Sampling for Large Language Models [27.807695570974644]
  Training models to achieve effective self-correction and self-correction remains a significant challenge. We propose a novel iterative sampling algorithm framework designed to enhance self-correction and generate higher-quality samples.
  arXiv  Detail & Related papers  (2025-02-08T04:39:51Z)
• Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains [114.76612918465948]
  Large language models (LLMs) have achieved remarkable performance in recent years but are fundamentally limited by the underlying training data. We propose a complementary approach towards self-improvement where finetuning is applied to a multiagent society of language models.
  arXiv  Detail & Related papers  (2025-01-10T04:35:46Z)
• Self-Improvement in Language Models: The Sharpening Mechanism [70.9248553790022]
  We offer a new perspective on the capabilities of self-improvement through a lens we refer to as sharpening. Motivated by the observation that language models are often better at verifying response quality than they are at generating correct responses, we formalize self-improvement as using the model itself as a verifier during post-training. We analyze two natural families of self-improvement algorithms based on SFT and RLHF.
  arXiv  Detail & Related papers  (2024-12-02T20:24:17Z)
• Can I understand what I create? Self-Knowledge Evaluation of Large Language Models [31.85129258347539]
  Large language models (LLMs) have achieved remarkable progress in linguistic tasks. Inspired by Feynman's principle of understanding through creation, we introduce a self-knowledge evaluation framework.
  arXiv  Detail & Related papers  (2024-06-10T09:53:54Z)
• LLMs can learn self-restraint through iterative self-reflection [57.26854891567574]
  Large Language Models (LLMs) must be capable of dynamically adapting their behavior based on their level of knowledge and uncertainty associated with specific topics. This adaptive behavior, which we refer to as self-restraint, is non-trivial to teach. We devise a utility function that can encourage the model to produce responses only when it is confident in them.
  arXiv  Detail & Related papers  (2024-05-15T13:35:43Z)
• SELF: Self-Evolution with Language Feedback [68.6673019284853]
  'SELF' (Self-Evolution with Language Feedback) is a novel approach to advance large language models. It enables LLMs to self-improve through self-reflection, akin to human learning processes. Our experiments in mathematics and general tasks demonstrate that SELF can enhance the capabilities of LLMs without human intervention.
  arXiv  Detail & Related papers  (2023-10-01T00:52:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.