Automatic Chain of Thought Prompting in Large Language Models

• URL: http://arxiv.org/abs/2210.03493v1
• Date: Fri, 7 Oct 2022 12:28:21 GMT
• Title: Automatic Chain of Thought Prompting in Large Language Models
• Authors: Zhuosheng Zhang, Aston Zhang, Mu Li, Alex Smola
• Abstract summary: Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. "Let's think step by step" prompt generates reasoning chains for demonstrations one by one. We propose an automatic CoT prompting method: Auto-CoT.
• Score: 20.54898481696753
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. Providing these steps for prompting demonstrations is called chain-of-thought (CoT) prompting. CoT prompting has two major paradigms. One leverages a simple prompt like "Let's think step by step" to facilitate step-by-step thinking before answering a question. The other uses a few manual demonstrations one by one, each composed of a question and a reasoning chain that leads to an answer. The superior performance of the second paradigm hinges on the hand-crafting of task-specific demonstrations one by one. We show that such manual efforts may be eliminated by leveraging LLMs with the "Let's think step by step" prompt to generate reasoning chains for demonstrations one by one, i.e., let's think not just step by step, but also one by one. However, these generated chains often come with mistakes. To mitigate the effect of such mistakes, we find that diversity matters for automatically constructing demonstrations. We propose an automatic CoT prompting method: Auto-CoT. It samples questions with diversity and generates reasoning chains to construct demonstrations. On ten public benchmark reasoning tasks with GPT-3, Auto-CoT consistently matches or exceeds the performance of the CoT paradigm that requires manual designs of demonstrations. Code is available at https://github.com/amazon-research/auto-cot

Related papers

• Watch Your Steps: Observable and Modular Chains of Thought [36.79118554877861]
  We propose a variant of chain of thought (CoT) prompting called Program Trace Prompting. It makes explanations more observable while preserving the power, generality and flexibility of CoT. Program Trace Prompting is applicable to many tasks, achieving strong results on the 23 diverse tasks in the BIG-Bench Hard benchmark.
  arXiv  Detail & Related papers  (2024-09-17T23:47:20Z)
• Self-Harmonized Chain of Thought [8.540320749424172]
  Chain-of-Thought (CoT) prompting reveals that large language models are capable of performing complex reasoning via intermediate steps. ECHO demonstrates the best overall performance across three reasoning domains.
  arXiv  Detail & Related papers  (2024-09-06T06:57:04Z)
• Pattern-Aware Chain-of-Thought Prompting in Large Language Models [26.641713417293538]
  Chain-of-thought (CoT) prompting can guide language models to engage in complex multi-step reasoning. We show that the underlying reasoning patterns play a more crucial role in such tasks. We propose Pattern-Aware CoT, a prompting method that considers the diversity of demonstration patterns.
  arXiv  Detail & Related papers  (2024-04-23T07:50:00Z)
• Generalizable Chain-of-Thought Prompting in Mixed-task Scenarios with Large Language Models [68.05046964022844]
  Large language models (LLMs) have unveiled remarkable reasoning capabilities by exploiting chain-of-thought (CoT) prompting. We propose GeM-CoT, a Generalizable CoT prompting mechanism in Mixed-task scenarios where the type of input questions is unknown. With this technical design, GeM-CoT simultaneously enjoys superior generalization capabilities and remarkable performances on 10 public reasoning tasks and 23 BBH tasks.
  arXiv  Detail & Related papers  (2023-10-10T15:10:03Z)
• Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Step [133.60124577507727]
  Chain-of-thought prompting primes large language models to verbalize rationalization for their predictions. We show that orders-of-magnitude smaller models can still benefit from chain-of-thought prompting. We introduce Chain-of-Thought Distillation (SCoTD), a method to train a smaller student model on rationalizations sampled from a significantly larger teacher model.
  arXiv  Detail & Related papers  (2023-06-24T20:15:07Z)
• Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models [81.01397924280612]
  Large language models (LLMs) can achieve highly effective performance on various reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting as demonstrations. We introduce Iter-CoT (Iterative bootstrapping in Chain-of-Thoughts Prompting), an iterative bootstrapping approach for selecting exemplars and generating reasoning chains.
  arXiv  Detail & Related papers  (2023-04-23T13:54:39Z)
• Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models [121.54462976635743]
  Large language models can perform various reasoning tasks by using chain-of-thought prompting, which guides them to find answers through step-by-step demonstrations. We introduce Synthetic prompting, a method that leverages a few handcrafted examples to prompt the model to generate more examples by itself. We evaluate our method on numerical, symbolic, and algorithmic reasoning tasks, and show that it outperforms existing prompting techniques.
  arXiv  Detail & Related papers  (2023-02-01T17:33:12Z)
• Complexity-Based Prompting for Multi-Step Reasoning [72.0057198610614]
  We study the task of prompting large-scale language models to perform multi-step reasoning. A central question is which reasoning examples make the most effective prompts. We propose complexity-based prompting, a simple and effective example selection scheme for multi-step reasoning.
  arXiv  Detail & Related papers  (2022-10-03T05:33:27Z)
• Automatic Label Sequence Generation for Prompting Sequence-to-sequence Models [105.4590533269863]
  We propose AutoSeq, a fully automatic prompting method. We adopt natural language prompts on sequence-to-sequence models. Our method reveals the potential of sequence-to-sequence models in few-shot learning.
  arXiv  Detail & Related papers  (2022-09-20T01:35:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.