DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

• URL: http://arxiv.org/abs/2310.16436v2
• Date: Thu, 26 Oct 2023 04:16:52 GMT
• Title: DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models
• Authors: Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, Sibei Yang
• Abstract summary: Large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. The transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation. This study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning.
• Score: 28.712359821231182
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A long-standing goal of AI systems is to perform complex multimodal reasoning like humans. Recently, large language models (LLMs) have made remarkable strides in such multi-step reasoning on the language modality solely by leveraging the chain of thought (CoT) to mimic human thinking. However, the transfer of these advancements to multimodal contexts introduces heightened challenges, including but not limited to the impractical need for labor-intensive annotation and the limitations in terms of flexibility, generalizability, and explainability. To evoke CoT reasoning in multimodality, this work first conducts an in-depth analysis of these challenges posed by multimodality and presents two key insights: "keeping critical thinking" and "letting everyone do their jobs" in multimodal CoT reasoning. Furthermore, this study proposes a novel DDCoT prompting that maintains a critical attitude through negative-space prompting and incorporates multimodality into reasoning by first dividing the reasoning responsibility of LLMs into reasoning and recognition and then integrating the visual recognition capability of visual models into the joint reasoning process. The rationales generated by DDCoT not only improve the reasoning abilities of both large and small language models in zero-shot prompting and fine-tuning learning, significantly outperforming state-of-the-art methods but also exhibit impressive generalizability and explainability.

Related papers

• Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models [64.1799100754406]
  Large Language Models (LLMs) demonstrate enhanced capabilities and reliability by reasoning more. Despite various efforts to improve LLM reasoning, high-quality long-chain reasoning data and optimized training pipelines still remain inadequately explored in vision-language tasks. We present Insight-V, an early effort to 1) scalably produce long and robust reasoning data for complex multi-modal tasks, and 2) an effective training pipeline to enhance the reasoning capabilities of MLLMs.
  arXiv  Detail & Related papers  (2024-11-21T18:59:55Z)
• Cantor: Inspiring Multimodal Chain-of-Thought of MLLM [83.6663322930814]
  We argue that converging visual context acquisition and logical reasoning is pivotal for tackling visual reasoning tasks. We propose an innovative multimodal CoT framework, termed Cantor, characterized by a perception-decision architecture. Our experiments demonstrate the efficacy of the proposed framework, showing significant improvements in multimodal CoT performance.
  arXiv  Detail & Related papers  (2024-04-24T17:59:48Z)
• Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models [25.058162782167503]
  Chain-of-thought (CoT) reasoning has exhibited impressive performance in language models for solving complex tasks and answering questions. We introduce a novel approach for multi-modal CoT reasoning that utilizes latent space learning via diffusion processes to generate effective image features that align with language thoughts. Our method fuses image features and text representations at a deep level and improves the complex reasoning ability of multi-modal CoT.
  arXiv  Detail & Related papers  (2023-12-14T09:13:09Z)
• Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agents [80.5213198675411]
  Large language models (LLMs) have dramatically enhanced the field of language intelligence. LLMs leverage the intriguing chain-of-thought (CoT) reasoning techniques, obliging them to formulate intermediate steps en route to deriving an answer. Recent research endeavors have extended CoT reasoning methodologies to nurture the development of autonomous language agents.
  arXiv  Detail & Related papers  (2023-11-20T14:30:55Z)
• Towards CausalGPT: A Multi-Agent Approach for Faithful Knowledge Reasoning via Promoting Causal Consistency in LLMs [60.244412212130264]
  Causal-Consistency Chain-of-Thought harnesses multi-agent collaboration to bolster the faithfulness and causality of foundation models. Our framework demonstrates significant superiority over state-of-the-art methods through extensive and comprehensive evaluations.
  arXiv  Detail & Related papers  (2023-08-23T04:59:21Z)
• Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction [8.169359626365619]
  We generate a textitchain of thought (CoT) -- a sequence of intermediate reasoning steps. We present a novel conditional prompt distillation method to assimilate the commonsense reasoning ability from large language models. Our approach attains state-of-the-art accuracy and manifests a plethora of advantages concerning interpretability, data efficiency, and cross-domain generalization.
  arXiv  Detail & Related papers  (2023-06-25T04:33:56Z)
• Large Language Models are In-Context Semantic Reasoners rather than Symbolic Reasoners [75.85554779782048]
  Large Language Models (LLMs) have excited the natural language and machine learning community over recent years. Despite of numerous successful applications, the underlying mechanism of such in-context capabilities still remains unclear. In this work, we hypothesize that the learned textitsemantics of language tokens do the most heavy lifting during the reasoning process.
  arXiv  Detail & Related papers  (2023-05-24T07:33:34Z)
• Visual Chain of Thought: Bridging Logical Gaps with Multimodal Infillings [61.04460792203266]
  We introduce VCoT, a novel method that leverages chain-of-thought prompting with vision-language grounding to bridge the logical gaps within sequential data. Our method uses visual guidance to generate synthetic multimodal infillings that add consistent and novel information to reduce the logical gaps for downstream tasks.
  arXiv  Detail & Related papers  (2023-05-03T17:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.