Patience Is The Key to Large Language Model Reasoning

• URL: http://arxiv.org/abs/2411.13082v3
• Date: Wed, 04 Dec 2024 07:22:45 GMT
• Title: Patience Is The Key to Large Language Model Reasoning
• Authors: Yijiong Yu,
• Abstract summary: We propose a simple method by encouraging models to adopt a more patient reasoning style. We generate detailed reasoning processes as positive examples and simple answers as negative examples, thereby training the model to favor thoroughness in its responses. Our results demonstrate a performance increase of up to 2.1% on GSM8k with training just on a lightweight dataset.
• Score: 0.0
• License:
• Abstract: Recent advancements in the field of large language models, particularly through the Chain of Thought (CoT) approach, have demonstrated significant improvements in solving complex problems. However, existing models either tend to sacrifice detailed reasoning for brevity due to user preferences, or require extensive and expensive training data to learn complicated reasoning ability, limiting their potential in solving complex tasks. To bridge this gap, following the concept of scaling test-time, we propose a simple method by encouraging models to adopt a more patient reasoning style without the need of introducing new knowledge or skills. To employ a preference optimization approach, we generate detailed reasoning processes as positive examples and simple answers as negative examples, thereby training the model to favor thoroughness in its responses. Our results demonstrate a performance increase of up to 2.1% on GSM8k with training just on a lightweight dataset.

Related papers

• Iterative Deepening Sampling for Large Language Models [27.807695570974644]
  Training models to achieve effective self-correction and self-correction remains a significant challenge. We propose a novel iterative sampling algorithm framework designed to enhance self-correction and generate higher-quality samples.
  arXiv  Detail & Related papers  (2025-02-08T04:39:51Z)
• Training Language Models to Reason Efficiently [14.390800014819439]
  We use reinforcement learning to train large reasoning models to reason efficiently. Our method incentivizes models to minimize unnecessary computational overhead while maintaining accuracy. Experiments on two open-weight large reasoning models demonstrate significant reductions in inference cost while preserving most of the accuracy.
  arXiv  Detail & Related papers  (2025-02-06T19:18:16Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)
• Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression [64.07696663255155]
  Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. We introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT) which effectively transfers the knowledge of LLMs to extremely small-scale models.
  arXiv  Detail & Related papers  (2023-10-24T07:58:20Z)
• An Emulator for Fine-Tuning Large Language Models using Small Language Models [91.02498576056057]
  We introduce emulated fine-tuning (EFT), a principled and practical method for sampling from a distribution that approximates the result of pre-training and fine-tuning at different scales. We show that EFT enables test-time adjustment of competing behavioral traits like helpfulness and harmlessness without additional training. Finally, a special case of emulated fine-tuning, which we call LM up-scaling, avoids resource-intensive fine-tuning of large pre-trained models by ensembling them with small fine-tuned models.
  arXiv  Detail & Related papers  (2023-10-19T17:57:16Z)
• RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models [57.12888828853409]
  RAVEN is a model that combines retrieval-augmented masked language modeling and prefix language modeling. Fusion-in-Context Learning enables the model to leverage more in-context examples without requiring additional training. Our work underscores the potential of retrieval-augmented encoder-decoder language models for in-context learning.
  arXiv  Detail & Related papers  (2023-08-15T17:59:18Z)
• Teaching Smaller Language Models To Generalise To Unseen Compositional Questions [6.9076450524134145]
  We propose a combination of multitask pretraining on up to 93 tasks designed to instill diverse reasoning abilities. We show that performance can be significantly improved by adding retrieval-augmented training datasets.
  arXiv  Detail & Related papers  (2023-08-02T05:00:12Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Few-shot Prompting Towards Controllable Response Generation [49.479958672988566]
  We first explored the combination of prompting and reinforcement learning (RL) to steer models' generation without accessing any of the models' parameters. We apply multi-task learning to make the model learn to generalize to new tasks better. Experiment results show that our proposed method can successfully control several state-of-the-art (SOTA) dialogue models without accessing their parameters.
  arXiv  Detail & Related papers  (2022-06-08T14:48:06Z)
• Building Accurate Simple Models with Multihop [13.182955266765653]
  We propose a meta-approach where we transfer information from the complex model to the simple model. Our approach can transfer information between consecutive models in the sequence using any of the previously mentioned approaches. In the experiments on real data, we observe that we get consistent gains for different choices of models over 1-hop.
  arXiv  Detail & Related papers  (2021-09-14T20:39:11Z)
• Incremental Learning for Personalized Recommender Systems [8.020546404087922]
  We present an incremental learning solution to provide both the training efficiency and the model quality. The solution is deployed in LinkedIn and directly applicable to industrial scale recommender systems.
  arXiv  Detail & Related papers  (2021-08-13T04:21:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.