Abstraction-of-Thought Makes Language Models Better Reasoners

• URL: http://arxiv.org/abs/2406.12442v2
• Date: Thu, 26 Sep 2024 11:15:14 GMT
• Title: Abstraction-of-Thought Makes Language Models Better Reasoners
• Authors: Ruixin Hong, Hongming Zhang, Xiaoman Pan, Dong Yu, Changshui Zhang,
• Abstract summary: We introduce a novel structured reasoning format called Abstraction-of-Thought (AoT) The uniqueness of AoT lies in its explicit requirement for varying levels of abstraction within the reasoning process. We present AoT Collection, a generic finetuning dataset consisting of 348k high-quality samples with AoT reasoning processes.
• Score: 79.72672444664376
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Abstract reasoning, the ability to reason from the abstract essence of a problem, serves as a key to generalization in human reasoning. However, eliciting language models to perform reasoning with abstraction remains unexplored. This paper seeks to bridge this gap by introducing a novel structured reasoning format called Abstraction-of-Thought (AoT). The uniqueness of AoT lies in its explicit requirement for varying levels of abstraction within the reasoning process. This approach could elicit language models to first contemplate on the abstract level before incorporating concrete details, which is overlooked by the prevailing step-by-step Chain-of-Thought (CoT) method. To align models with the AoT format, we present AoT Collection, a generic finetuning dataset consisting of 348k high-quality samples with AoT reasoning processes, collected via an automated and scalable pipeline. We finetune a wide range of language models with AoT Collection and conduct extensive evaluations on 23 unseen tasks from the challenging benchmark Big-Bench Hard. Experimental results indicate that models aligned to AoT reasoning format substantially outperform those aligned to CoT in many reasoning tasks.

Related papers

• Conceptual and Unbiased Reasoning in Language Models [98.90677711523645]
  We propose a novel conceptualization framework that forces models to perform conceptual reasoning on abstract questions. We show that existing large language models fall short on conceptual reasoning, dropping 9% to 28% on various benchmarks. We then discuss how models can improve since high-level abstract reasoning is key to unbiased and generalizable decision-making.
  arXiv  Detail & Related papers  (2024-03-30T00:53:53Z)
• Large Language Models as Analogical Reasoners [155.9617224350088]
  Chain-of-thought (CoT) prompting for language models demonstrates impressive performance across reasoning tasks. We introduce a new prompting approach, analogical prompting, designed to automatically guide the reasoning process of large language models.
  arXiv  Detail & Related papers  (2023-10-03T00:57:26Z)
• Deep Non-Monotonic Reasoning for Visual Abstract Reasoning Tasks [3.486683381782259]
  This paper proposes a non-monotonic computational approach to solve visual abstract reasoning tasks. We implement a deep learning model using this approach and tested it on the RAVEN dataset -- a dataset inspired by the Raven's Progressive Matrices test.
  arXiv  Detail & Related papers  (2023-02-08T16:35:05Z)
• Learning to Reason With Relational Abstractions [65.89553417442049]
  We study how to build stronger reasoning capability in language models using the idea of relational abstractions. We find that models that are supplied with such sequences as prompts can solve tasks with a significantly higher accuracy.
  arXiv  Detail & Related papers  (2022-10-06T00:27:50Z)
• Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango [11.344587937052697]
  This work initiates the preliminary steps towards a deeper understanding of reasoning mechanisms in large language models. Our work centers around querying the model while controlling for all but one of the components in a prompt: symbols, patterns, and text. We posit that text imbues patterns with commonsense knowledge and meaning.
  arXiv  Detail & Related papers  (2022-09-16T02:54:00Z)
• Chain of Thought Prompting Elicits Reasoning in Large Language Models [56.811278668446825]
  This paper explores the ability of language models to generate a coherent chain of thought. Experiments show that inducing a chain of thought via prompting can enable sufficiently large language models to better perform reasoning tasks.
  arXiv  Detail & Related papers  (2022-01-28T02:33:07Z)
• Does entity abstraction help generative Transformers reason? [8.159805544989359]
  We study the utility of incorporating entity type abstractions into pre-trained Transformers. We test these methods on four NLP tasks requiring different forms of logical reasoning.
  arXiv  Detail & Related papers  (2022-01-05T19:00:53Z)
• To Point or Not to Point: Understanding How Abstractive Summarizers Paraphrase Text [4.4044968357361745]
  We characterize how one popular abstractive model, the pointer-generator model of See et al., uses its explicit copy/generation switch to control its level of abstraction. When we modify the copy/generation switch and force the model to generate, only simple neural abilities are revealed alongside factual inaccuracies and hallucinations. In line with previous research, these results suggest that abstractive summarization models lack the semantic understanding necessary to generate paraphrases that are both abstractive and faithful to the source document.
  arXiv  Detail & Related papers  (2021-06-03T04:03:15Z)
• Discrete Reasoning Templates for Natural Language Understanding [79.07883990966077]
  We present an approach that reasons about complex questions by decomposing them to simpler subquestions. We derive the final answer according to instructions in a predefined reasoning template. We show that our approach is competitive with the state-of-the-art while being interpretable and requires little supervision.
  arXiv  Detail & Related papers  (2021-04-05T18:56:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.