Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models

• URL: http://arxiv.org/abs/2410.08436v1
• Date: Fri, 11 Oct 2024 00:45:50 GMT
• Title: Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models
• Authors: Zi'ou Zheng, Christopher Malon, Martin Renqiang Min, Xiaodan Zhu,
• Abstract summary: This paper is centred around a focused study: whether the current state-of-the-art generalist LLMs can leverage the structures in a few examples to better construct the proof structures with textitin-context learning.
• Score: 30.09120709652445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When performing complex multi-step reasoning tasks, the ability of Large Language Models (LLMs) to derive structured intermediate proof steps is important for ensuring that the models truly perform the desired reasoning and for improving models' explainability. This paper is centred around a focused study: whether the current state-of-the-art generalist LLMs can leverage the structures in a few examples to better construct the proof structures with \textit{in-context learning}. Our study specifically focuses on structure-aware demonstration and structure-aware pruning. We demonstrate that they both help improve performance. A detailed analysis is provided to help understand the results.

Related papers

• Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning [52.83539473110143]
  We introduce a novel structure-oriented analysis method to help Large Language Models (LLMs) better understand a question. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA) Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods.
  arXiv  Detail & Related papers  (2024-10-18T05:30:33Z)
• Large Language Model-driven Meta-structure Discovery in Heterogeneous Information Network [29.149367323751413]
  We propose ReStruct, a meta-structure search framework that integrates reasoning into the evolutionary procedure. We show that ReStruct achieves state-of-the-art performance in both recommendation and node classification tasks.
  arXiv  Detail & Related papers  (2024-02-18T09:21:12Z)
• Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning [14.219239732584368]
  In this work, we investigate constructing and leveraging extracted semantic structures (graphs) for multi-hop question answering. Empirical results and human evaluations show that our framework: generates more faithful reasoning chains and substantially improves the QA performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2023-11-07T05:32:39Z)
• "You Are An Expert Linguistic Annotator": Limits of LLMs as Analyzers of Abstract Meaning Representation [60.863629647985526]
  We examine the successes and limitations of the GPT-3, ChatGPT, and GPT-4 models in analysis of sentence meaning structure. We find that models can reliably reproduce the basic format of AMR, and can often capture core event, argument, and modifier structure. Overall, our findings indicate that these models out-of-the-box can capture aspects of semantic structure, but there remain key limitations in their ability to support fully accurate semantic analyses or parses.
  arXiv  Detail & Related papers  (2023-10-26T21:47:59Z)
• Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure [66.33623392497599]
  We show that a structure called template-content structure (T-C structure) can reduce the possible space from exponential level to linear level. We demonstrate that models can achieve task composition, further reducing the space needed to learn from linear to logarithmic.
  arXiv  Detail & Related papers  (2023-10-09T06:57:45Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings is a significant challenge. We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences. We demonstrate that generative models like GPT can accurately learn this CFG language and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• StructGPT: A General Framework for Large Language Model to Reason over Structured Data [117.13986738340027]
  We develop an emphIterative Reading-then-Reasoning(IRR) approach for solving question answering tasks based on structured data. Our approach can significantly boost the performance of ChatGPT and achieve comparable performance against the full-data supervised-tuning baselines.
  arXiv  Detail & Related papers  (2023-05-16T17:45:23Z)
• Explaining Large Language Model-Based Neural Semantic Parsers (Student Abstract) [0.0]
  Large language models (LLMs) have demonstrated strong capability in structured prediction tasks such as semantic parsing. Our work studies different methods for explaining an LLM-based semantic semantic behaviors. We hope to inspire future research toward better understanding them.
  arXiv  Detail & Related papers  (2023-01-25T16:12:43Z)
• Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoning [26.811507121199323]
  This paper proposes a unified learning framework that combines explicit structure reasoning and language pre-training to endow PLMs with the structure reasoning skill. It first identifies several elementary structures within contexts to construct structured queries and performs step-by-step reasoning along the queries to identify the answer entity. Experimental results on four datasets demonstrate that the proposed model achieves significant improvements in complex reasoning tasks involving diverse structures.
  arXiv  Detail & Related papers  (2023-01-21T08:18:11Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.