Forward-Backward Reasoning in Large Language Models for Mathematical Verification

• URL: http://arxiv.org/abs/2308.07758v6
• Date: Wed, 5 Jun 2024 03:37:35 GMT
• Title: Forward-Backward Reasoning in Large Language Models for Mathematical Verification
• Authors: Weisen Jiang, Han Shi, Longhui Yu, Zhengying Liu, Yu Zhang, Zhenguo Li, James T. Kwok,
• Abstract summary: Self-Consistency samples diverse reasoning chains with answers and chooses the final answer by majority voting. We introduce backward reasoning to verify candidate answers. FOrward and BAckward Reasoning for verification achieves state-of-the-art performance.
• Score: 65.9495774606273
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-Consistency samples diverse reasoning chains with answers and chooses the final answer by majority voting. It is based on forward reasoning and cannot further improve performance by sampling more reasoning chains when saturated. To further boost performance, we introduce backward reasoning to verify candidate answers. Specifically, for mathematical tasks, we mask a number in the question and ask the LLM to answer a backward question created by a simple template, i.e., to predict the masked number when a candidate answer is provided. Instead of using forward or backward reasoning alone, we propose FOBAR to combine FOrward and BAckward Reasoning for verification. Extensive experiments on six standard mathematical data sets and three LLMs show that FOBAR achieves state-of-the-art performance. In particular, FOBAR outperforms Self-Consistency, which uses forward reasoning alone, demonstrating that combining forward and forward reasoning is better. In addition, FOBAR performs better than existing verification methods, showing the effectiveness of the simple template used in backward reasoning and the proposed combination. Extensions to non-mathematical problems are also discussed and validated empirically.

Related papers

• Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models [84.15513004135576]
  Current research enhances the reasoning performance of Large Language Models (LLMs) by sampling multiple reasoning chains and ensembling based on the answer frequency. This approach fails in scenarios where the correct answers are in the minority. We introduce a hierarchical reasoning aggregation framework AoR, which selects answers based on the evaluation of reasoning chains.
  arXiv  Detail & Related papers  (2024-05-21T17:12:19Z)
• Premise Order Matters in Reasoning with Large Language Models [57.18850969634412]
  We show that large language models (LLMs) are surprisingly brittle to the ordering of the premises. We observe that LLMs achieve the best performance when the premise order aligns with the context required in intermediate reasoning steps.
  arXiv  Detail & Related papers  (2024-02-14T04:50:18Z)
• On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks [17.329365493094542]
  We present a principled empirical study of the performance of GPT-4 in three domains: Game of 24, Graph Coloring, and STRIPS planning. We observe significant performance collapse with self-critique and significant performance gains with sound external verification.
  arXiv  Detail & Related papers  (2024-02-12T23:11:01Z)
• Question Decomposition Improves the Faithfulness of Model-Generated Reasoning [23.34325378824462]
  Large language models (LLMs) are difficult to verify the correctness and safety of their behavior. One approach is to prompt LLMs to externalize their reasoning, by having them generate step-by-step reasoning as they answer a question. This approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT.
  arXiv  Detail & Related papers  (2023-07-17T00:54:10Z)
• Large Language Models are Better Reasoners with Self-Verification [48.534270563880845]
  Large language models (LLMs) have shown strong reasoning ability in several natural language processing tasks. LLMs with chain of thought (CoT) prompting require multi-step prompting and multi-token prediction, which is highly sensitive to individual mistakes. We propose and prove that LLMs also have similar self-verification abilities.
  arXiv  Detail & Related papers  (2022-12-19T15:51:52Z)
• Complexity-Based Prompting for Multi-Step Reasoning [72.0057198610614]
  We study the task of prompting large-scale language models to perform multi-step reasoning. A central question is which reasoning examples make the most effective prompts. We propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.
  arXiv  Detail & Related papers  (2022-10-03T05:33:27Z)
• Faithful Reasoning Using Large Language Models [12.132449274592668]
  We show how LMs can be made to perform faithful multi-step reasoning via a process whose causal structure mirrors the underlying logical structure of the problem. Our approach works by chaining together reasoning steps, where each step results from calls to two fine-tuned LMs. We demonstrate the effectiveness of our model on multi-step logical deduction and scientific question-answering, showing that it outperforms baselines on final answer accuracy.
  arXiv  Detail & Related papers  (2022-08-30T13:44:41Z)
• Generative Context Pair Selection for Multi-hop Question Answering [60.74354009152721]
  We propose a generative context selection model for multi-hop question answering. Our proposed generative passage selection model has a better performance (4.9% higher than baseline) on adversarial held-out set.
  arXiv  Detail & Related papers  (2021-04-18T07:00:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.