Watch Your Steps: Observable and Modular Chains of Thought

• URL: http://arxiv.org/abs/2409.15359v2
• Date: Tue, 1 Oct 2024 20:24:38 GMT
• Title: Watch Your Steps: Observable and Modular Chains of Thought
• Authors: Cassandra A. Cohen, William W. Cohen,
• Abstract summary: We propose a variant of chain of thought (CoT) prompting called Program Trace Prompting. It makes explanations more observable while preserving the power, generality and flexibility of CoT. Program Trace Prompting is applicable to many tasks, achieving strong results on the 23 diverse tasks in the BIG-Bench Hard benchmark.
• Score: 36.79118554877861
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a variant of chain of thought (CoT) prompting called Program Trace Prompting that makes explanations more observable while preserving the power, generality and flexibility of CoT. In our approach, few-shot CoT demonstrations are wrapped in a formal syntax based on Python, and each prompt: identifies and names steps; defines the input/output behavior of steps; and replaces CoT explanations of in-context examples with chains of these formalized steps on the same examples. Program Trace Prompting is applicable to many tasks, achieving strong results on the 23 diverse tasks in the BIG-Bench Hard benchmark. More importantly, by instrumenting explanations in this way, we enable new types of analysis. In particular, we identify "non-local errors" (which correspond to incorrectly learning the reasoning method illustrated in the demonstrations) as an unaddressed issue in CoT learning, and we present methods for verifying the modularity of steps in a CoT explanation.

Related papers

• Pattern-Aware Chain-of-Thought Prompting in Large Language Models [26.641713417293538]
  Chain-of-thought (CoT) prompting can guide language models to engage in complex multi-step reasoning. We show that the underlying reasoning patterns play a more crucial role in such tasks. We propose Pattern-Aware CoT, a prompting method that considers the diversity of demonstration patterns.
  arXiv  Detail & Related papers  (2024-04-23T07:50:00Z)
• Contrastive Instruction Tuning [61.97704869248903]
  We propose Contrastive Instruction Tuning to maximize the similarity between semantically equivalent instruction-instance pairs. Experiments on the PromptBench benchmark show that CoIN consistently improves LLMs' robustness to unseen instructions with variations across character, word, sentence, and semantic levels by an average of +2.5% in accuracy.
  arXiv  Detail & Related papers  (2024-02-17T00:09:32Z)
• Tab-CoT: Zero-shot Tabular Chain of Thought [7.558415495951758]
  We propose Tab-CoT, which allows the complex reasoning process to be explicitly modelled in a highly structured manner. Despite its simplicity, we show that our approach is capable of performing reasoning across multiple dimensions. We demonstrate our approach's strong zero-shot and few-shot capabilities through extensive experiments on a range of reasoning tasks.
  arXiv  Detail & Related papers  (2023-05-28T20:49:52Z)
• Enhancing Chain-of-Thoughts Prompting with Iterative Bootstrapping in Large Language Models [81.01397924280612]
  Large language models (LLMs) can achieve highly effective performance on various reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting as demonstrations. We introduce Iter-CoT (Iterative bootstrapping in Chain-of-Thoughts Prompting), an iterative bootstrapping approach for selecting exemplars and generating reasoning chains.
  arXiv  Detail & Related papers  (2023-04-23T13:54:39Z)
• Explanation Selection Using Unlabeled Data for Chain-of-Thought Prompting [80.9896041501715]
  Explanations that have not been "tuned" for a task, such as off-the-shelf explanations written by nonexperts, may lead to mediocre performance. This paper tackles the problem of how to optimize explanation-infused prompts in a blackbox fashion.
  arXiv  Detail & Related papers  (2023-02-09T18:02:34Z)
• Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models [121.54462976635743]
  Large language models can perform various reasoning tasks by using chain-of-thought prompting, which guides them to find answers through step-by-step demonstrations. We introduce Synthetic prompting, a method that leverages a few handcrafted examples to prompt the model to generate more examples by itself. We evaluate our method on numerical, symbolic, and algorithmic reasoning tasks, and show that it outperforms existing prompting techniques.
  arXiv  Detail & Related papers  (2023-02-01T17:33:12Z)
• Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters [82.84696222087396]
  Chain-of-Thought (CoT) prompting can dramatically improve the multi-step reasoning abilities of large language models (LLMs) We show that CoT reasoning is possible even with invalid demonstrations.
  arXiv  Detail & Related papers  (2022-12-20T05:20:54Z)
• Automatic Chain of Thought Prompting in Large Language Models [20.54898481696753]
  Large language models (LLMs) can perform complex reasoning by generating intermediate reasoning steps. "Let's think step by step" prompt generates reasoning chains for demonstrations one by one. We propose an automatic CoT prompting method: Auto-CoT.
  arXiv  Detail & Related papers  (2022-10-07T12:28:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.