Improving Language Model Reasoning with Self-motivated Learning

• URL: http://arxiv.org/abs/2404.07017v3
• Date: Tue, 30 Apr 2024 14:38:59 GMT
• Title: Improving Language Model Reasoning with Self-motivated Learning
• Authors: Yunlong Feng, Yang Xu, Libo Qin, Yasheng Wang, Wanxiang Che,
• Abstract summary: textitSelf-motivated Learning framework motivates the model itself to automatically generate rationales on existing datasets. We train a reward model with the rank to evaluate the quality of rationales, and improve the performance of reasoning through reinforcement learning.
• Score: 60.779625789039486
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale high-quality training data is important for improving the performance of models. After trained with data that has rationales (reasoning steps), models gain reasoning capability. However, the dataset with high-quality rationales is relatively scarce due to the high annotation cost. To address this issue, we propose \textit{Self-motivated Learning} framework. The framework motivates the model itself to automatically generate rationales on existing datasets. Based on the inherent rank from correctness across multiple rationales, the model learns to generate better rationales, leading to higher reasoning capability. Specifically, we train a reward model with the rank to evaluate the quality of rationales, and improve the performance of reasoning through reinforcement learning. Experiment results of Llama2 7B on multiple reasoning datasets show that our method significantly improves the reasoning ability of models, even outperforming text-davinci-002 in some datasets.

Related papers

• Improve Vision Language Model Chain-of-thought Reasoning [86.83335752119741]
  Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial for improving interpretability and trustworthiness. We show that training VLM on short answers does not generalize well to reasoning tasks that require more detailed responses.
  arXiv  Detail & Related papers  (2024-10-21T17:00:06Z)
• Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF [79.98542868281471]
  Reinforcement Learning from Human Feedback (RLHF) is a technique that aligns language models closely with human-centric values. It is observed that the performance of the reward model degrades after one epoch of training, and optimizing too much against the learned reward model eventually hinders the true objective. This paper delves into these issues, leveraging the theoretical insights to design improved reward learning algorithm termed 'Iterative Data Smoothing' (IDS)
  arXiv  Detail & Related papers  (2024-01-29T17:43:42Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)
• Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Training [49.3242278912771]
  Multimodal reasoning is a challenging task that requires models to reason across multiple modalities to answer questions. Existing approaches have made progress by incorporating language and visual modalities into a two-stage reasoning framework. We propose MC-CoT, a self-consistency training strategy that generates multiple rationales and answers, subsequently selecting the most accurate through a voting process.
  arXiv  Detail & Related papers  (2023-11-23T17:09:48Z)
• QualEval: Qualitative Evaluation for Model Improvement [82.73561470966658]
  We propose QualEval, which augments quantitative scalar metrics with automated qualitative evaluation as a vehicle for model improvement. QualEval uses a powerful LLM reasoner and our novel flexible linear programming solver to generate human-readable insights. We demonstrate that leveraging its insights, for example, improves the absolute performance of the Llama 2 model by up to 15% points relative.
  arXiv  Detail & Related papers  (2023-11-06T00:21:44Z)
• Counterfactual Collaborative Reasoning [41.89113539041682]
  Causal reasoning and logical reasoning are two important types of reasoning abilities for human intelligence. We propose Counterfactual Collaborative Reasoning, which conducts counterfactual logic reasoning to improve the performance. Experiments on three real-world datasets show that CCR achieves better performance than non-augmented models and implicitly augmented models.
  arXiv  Detail & Related papers  (2023-06-30T23:01:10Z)
• Entailment as Robust Self-Learner [14.86757876218415]
  We design a prompting strategy that formulates a number of different NLU tasks as contextual entailment. We propose the Simple Pseudo-Label Editing (SimPLE) algorithm for better pseudo-labeling quality in self-training.
  arXiv  Detail & Related papers  (2023-05-26T18:41:23Z)
• RLBoost: Boosting Supervised Models using Deep Reinforcement Learning [0.0]
  We present RLBoost, an algorithm that uses deep reinforcement learning strategies to evaluate a particular dataset and obtain a model capable of estimating the quality of any new data. The results of the article show that this model obtains better and more stable results than other state-of-the-art algorithms such as LOO, DataShapley or DVRL.
  arXiv  Detail & Related papers  (2023-05-23T14:38:33Z)
• Does Self-Rationalization Improve Robustness to Spurious Correlations? [19.553357015260687]
  We ask whether training models to self-rationalize can aid in their learning to solve tasks for the right reasons. We evaluate robustness to spurious correlations in fine-tuned encoder-decoder and decoder-only models of six different sizes. We find that while self-rationalization can improve robustness to spurious correlations in low-resource settings, it tends to hurt robustness in higher-resource settings.
  arXiv  Detail & Related papers  (2022-10-24T19:54:57Z)
• What to Learn, and How: Toward Effective Learning from Rationales [10.287185780246247]
  Learning from rationales seeks to augment model training with human-provided rationales that justify those labels. Our work highlights the importance of understanding properties of human explanations and exploiting them accordingly in model training.
  arXiv  Detail & Related papers  (2021-11-30T20:09:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.