Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning

• URL: http://arxiv.org/abs/2402.14963v2
• Date: Mon, 24 Jun 2024 10:05:24 GMT
• Title: Mirror: A Multiple-perspective Self-Reflection Method for Knowledge-rich Reasoning
• Authors: Hanqi Yan, Qinglin Zhu, Xinyu Wang, Lin Gui, Yulan He,
• Abstract summary: Large language models (LLMs) struggle with knowledge-rich problems without access to external resources. We propose Mirror, a Multiple-perspective self-reflection method for knowledge-rich reasoning.
• Score: 18.5717357875955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Large language models (LLMs) have the capability to iteratively reflect on their own outputs, recent studies have observed their struggles with knowledge-rich problems without access to external resources. In addition to the inefficiency of LLMs in self-assessment, we also observe that LLMs struggle to revisit their predictions despite receiving explicit negative feedback. Therefore, We propose Mirror, a Multiple-perspective self-reflection method for knowledge-rich reasoning, to avoid getting stuck at a particular reflection iteration. Mirror enables LLMs to reflect from multiple-perspective clues, achieved through a heuristic interaction between a Navigator and a Reasoner. It guides agents toward diverse yet plausibly reliable reasoning trajectory without access to ground truth by encouraging (1) diversity of directions generated by Navigator and (2) agreement among strategically induced perturbations in responses generated by the Reasoner. The experiments on five reasoning datasets demonstrate that Mirror's superiority over several contemporary self-reflection approaches. Additionally, the ablation study studies clearly indicate that our strategies alleviate the aforementioned challenges.

Related papers

• Thought Space Explorer: Navigating and Expanding Thought Space for Large Language Model Reasoning [15.918115880403152]
  We design the Thought Space Explorer (TSE) to expand and optimize thought structures to guide large language models (LLMs) to explore their blind spots of thinking. By generating new reasoning steps and branches based on the original thought structure with various designed strategies, TSE broadens the thought space and alleviates the impact of blind spots for LLM reasoning.
  arXiv  Detail & Related papers  (2024-10-31T17:12:14Z)
• Supporting Self-Reflection at Scale with Large Language Models: Insights from Randomized Field Experiments in Classrooms [7.550701021850185]
  We investigate the potential of Large Language Models (LLMs) to help students engage in post-lesson reflection. We conducted two randomized field experiments in undergraduate computer science courses.
  arXiv  Detail & Related papers  (2024-06-01T02:41:59Z)
• Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflection [90.71323430635593]
  We propose a novel self-detection paradigm that considers the comprehensive answer space beyond LLM-generated answers. Building upon this paradigm, we introduce a two-step framework, which firstly instructs LLM to reflect and provide justifications for each candidate answer. This framework can be seamlessly integrated with existing approaches for superior self-detection.
  arXiv  Detail & Related papers  (2024-03-15T02:38:26Z)
• How Far Are We from Intelligent Visual Deductive Reasoning? [41.4377002379162]
  We dig into vision-based deductive reasoning, a more sophisticated but less explored realm. We find previously unexposed blindspots in the current SOTA VLMs. We find that certain standard strategies that are effective when applied to LLMs do not seamlessly translate to the challenges presented by visual reasoning tasks.
  arXiv  Detail & Related papers  (2024-03-07T18:35:54Z)
• LEMMA: Towards LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation [58.524237916836164]
  We propose LEMMA: LVLM-Enhanced Multimodal Misinformation Detection with External Knowledge Augmentation. Our method improves the accuracy over the top baseline LVLM by 7% and 13% on Twitter and Fakeddit datasets respectively.
  arXiv  Detail & Related papers  (2024-02-19T08:32:27Z)
• Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives [45.87069217634753]
  Research indicates without external feedback, Large Language Model's intrinsic reflection is unstable. Our investigation unveils that the key bottleneck is the quality of the self-evaluated feedback. We advocate Self-Contrast: It adaptively explores diverse solving perspectives tailored to the request, contrasts the differences, and summarizes these discrepancies into a checklist which could be used to re-examine and eliminate discrepancies.
  arXiv  Detail & Related papers  (2024-01-04T00:32:33Z)
• The ART of LLM Refinement: Ask, Refine, and Trust [85.75059530612882]
  We propose a reasoning with refinement objective called ART: Ask, Refine, and Trust. It asks necessary questions to decide when an LLM should refine its output. It achieves a performance gain of +5 points over self-refinement baselines.
  arXiv  Detail & Related papers  (2023-11-14T07:26:32Z)
• Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection [74.51523859064802]
  We introduce a new framework called Self-Reflective Retrieval-Augmented Generation (Self-RAG) Self-RAG enhances an LM's quality and factuality through retrieval and self-reflection. It significantly outperforms state-of-the-art LLMs and retrieval-augmented models on a diverse set of tasks.
  arXiv  Detail & Related papers  (2023-10-17T18:18:32Z)
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate [85.3444184685235]
  We propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution. Our framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation.
  arXiv  Detail & Related papers  (2023-05-30T15:25:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.