ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning

• URL: http://arxiv.org/abs/2502.04689v2
• Date: Wed, 12 Feb 2025 18:36:24 GMT
• Title: ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning
• Authors: Yuwei Yin, Giuseppe Carenini,
• Abstract summary: Large language models (LLMs) achieve remarkable performance on challenging benchmarks that are structured as multiple-choice question-answering (QA) tasks. This paper introduces ARR, an intuitive and effective zero-shot prompting method that explicitly incorporates three key steps in QA solving: analyzing the intent of the question, retrieving relevant information, and reasoning step by step.
• Score: 22.825527641316192
• License:
• Abstract: Large language models (LLMs) achieve remarkable performance on challenging benchmarks that are often structured as multiple-choice question-answering (QA) tasks. Zero-shot Chain-of-Thought (CoT) prompting enhances reasoning in LLMs but provides only vague and generic guidance ("think step by step"). This paper introduces ARR, an intuitive and effective zero-shot prompting method that explicitly incorporates three key steps in QA solving: analyzing the intent of the question, retrieving relevant information, and reasoning step by step. Comprehensive experiments across diverse and challenging QA tasks demonstrate that ARR consistently improves the Baseline (without ARR prompting) and outperforms CoT. Ablation and case studies further validate the positive contributions of each component: analyzing, retrieving, and reasoning. Notably, intent analysis plays a vital role in ARR. Additionally, extensive evaluations across various model sizes, LLM series, and generation settings solidify the effectiveness, robustness, and generalizability of ARR.

Related papers

• Elevating Legal LLM Responses: Harnessing Trainable Logical Structures and Semantic Knowledge with Legal Reasoning [19.477062052536887]
  We propose the Logical-Semantic Integration Model (LSIM), a supervised framework that bridges semantic and logical coherence. LSIM comprises three components: reinforcement learning predicts a structured fact-rule chain for each question, a trainable Deep Structured Semantic Model (DSSM) retrieves the most relevant candidate questions and in-answer learning generates the final answer. Our experiments on a real-world legal dataset QA-validated through both automated metrics and human evaluation-demonstrate that LSIM significantly enhances accuracy and reliability compared to existing methods.
  arXiv  Detail & Related papers  (2025-02-11T19:33:07Z)
• A Survey of Query Optimization in Large Language Models [10.255235456427037]
  RAG mitigates the limitations of Large Language Models by dynamically retrieving and leveraging up-to-date relevant information. QO has emerged as a critical element, playing a pivotal role in determining the effectiveness of RAG's retrieval stage.
  arXiv  Detail & Related papers  (2024-12-23T13:26:04Z)
• RAG-Star: Enhancing Deliberative Reasoning with Retrieval Augmented Verification and Refinement [85.08223786819532]
  Existing large language models (LLMs) show exceptional problem-solving capabilities but might struggle with complex reasoning tasks. We propose textbfRAG-Star, a novel RAG approach that integrates retrieved information to guide the tree-based deliberative reasoning process. Our experiments involving Llama-3.1-8B-Instruct and GPT-4o demonstrate that RAG-Star significantly outperforms previous RAG and reasoning methods.
  arXiv  Detail & Related papers  (2024-12-17T13:05:36Z)
• Benchmarking and Understanding Compositional Relational Reasoning of LLMs [1.915591735124465]
  We first propose a new synthetic benchmark called Generalized Associative Recall (GAR) Evaluation shows that GAR is challenging enough for existing LLMs, revealing their fundamental deficiency in CRR. We then use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks and a set of vital attention heads.
  arXiv  Detail & Related papers  (2024-12-17T12:10:38Z)
• Make LLMs better zero-shot reasoners: Structure-orientated autonomous reasoning [52.83539473110143]
  We introduce a novel structure-oriented analysis method to help Large Language Models (LLMs) better understand a question. To further improve the reliability in complex question-answering tasks, we propose a multi-agent reasoning system, Structure-oriented Autonomous Reasoning Agents (SARA) Extensive experiments verify the effectiveness of the proposed reasoning system. Surprisingly, in some cases, the system even surpasses few-shot methods.
  arXiv  Detail & Related papers  (2024-10-18T05:30:33Z)
• Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks [68.49251303172674]
  State-of-the-art large language models (LLMs) exhibit impressive problem-solving capabilities but may struggle with complex reasoning and factual correctness. Existing methods harness the strengths of chain-of-thought and retrieval-augmented generation (RAG) to decompose a complex problem into simpler steps and apply retrieval to improve factual correctness. We introduce Critic-guided planning with Retrieval-augmentation, CR-Planner, a novel framework that leverages fine-tuned critic models to guide both reasoning and retrieval processes through planning.
  arXiv  Detail & Related papers  (2024-10-02T11:26:02Z)
• Benchmarking Large Language Models for Conversational Question Answering in Multi-instructional Documents [61.41316121093604]
  We present InsCoQA, a novel benchmark for evaluating large language models (LLMs) in the context of conversational question answering (CQA) Sourced from extensive, encyclopedia-style instructional content, InsCoQA assesses models on their ability to retrieve, interpret, and accurately summarize procedural guidance from multiple documents. We also propose InsEval, an LLM-assisted evaluator that measures the integrity and accuracy of generated responses and procedural instructions.
  arXiv  Detail & Related papers  (2024-10-01T09:10:00Z)
• Seek and Solve Reasoning for Table Question Answering [49.006950918895306]
  This paper reveals that the reasoning process during task simplification may be more valuable than the simplified tasks themselves. We propose a Seek-and-solving pipeline that instructs the LLM to first seek relevant information and then answer questions. We distill a single-step TQA-solving prompt from this pipeline, using demonstrations with SS-CoT paths to guide the LLM in solving complex TQA tasks.
  arXiv  Detail & Related papers  (2024-09-09T02:41:00Z)
• Aggregation of Reasoning: A Hierarchical Framework for Enhancing Answer Selection in Large Language Models [84.15513004135576]
  Current research enhances the reasoning performance of Large Language Models (LLMs) by sampling multiple reasoning chains and ensembling based on the answer frequency. This approach fails in scenarios where the correct answers are in the minority. We introduce a hierarchical reasoning aggregation framework AoR, which selects answers based on the evaluation of reasoning chains.
  arXiv  Detail & Related papers  (2024-05-21T17:12:19Z)
• Self-Convinced Prompting: Few-Shot Question Answering with Repeated Introspection [13.608076739368949]
  We introduce a novel framework that harnesses the potential of large-scale pre-trained language models. Our framework processes the output of a typical few-shot chain-of-thought prompt, assesses the correctness of the response, scrutinizes the answer, and ultimately produces a new solution.
  arXiv  Detail & Related papers  (2023-10-08T06:36:26Z)
• Active Prompting with Chain-of-Thought for Large Language Models [26.5029080638055]
  This paper proposes a new method, Active-Prompt, to adapt large language models to different tasks. By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty. Experimental results demonstrate the superiority of our proposed method, achieving state-of-the-art on eight complex reasoning tasks.
  arXiv  Detail & Related papers  (2023-02-23T18:58:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.