In-Context Ability Transfer for Question Decomposition in Complex QA
- URL: http://arxiv.org/abs/2310.18371v1
- Date: Thu, 26 Oct 2023 11:11:07 GMT
- Title: In-Context Ability Transfer for Question Decomposition in Complex QA
- Authors: Venktesh V, Sourangshu Bhattacharya, Avishek Anand
- Abstract summary: We propose icat (In-Context Ability Transfer) to solve complex question-answering tasks.
We transfer the ability to decompose complex questions to simpler questions or generate step-by-step rationales to LLMs.
We conduct large-scale experiments on a variety of complex QA tasks involving numerical reasoning, compositional complex QA, and heterogeneous complex QA.
- Score: 6.745884231594893
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Answering complex questions is a challenging task that requires question
decomposition and multistep reasoning for arriving at the solution. While
existing supervised and unsupervised approaches are specialized to a certain
task and involve training, recently proposed prompt-based approaches offer
generalizable solutions to tackle a wide variety of complex question-answering
(QA) tasks. However, existing prompt-based approaches that are effective for
complex QA tasks involve expensive hand annotations from experts in the form of
rationales and are not generalizable to newer complex QA scenarios and tasks.
We propose, icat (In-Context Ability Transfer) which induces reasoning
capabilities in LLMs without any LLM fine-tuning or manual annotation of
in-context samples. We transfer the ability to decompose complex questions to
simpler questions or generate step-by-step rationales to LLMs, by careful
selection from available data sources of related tasks. We also propose an
automated uncertainty-aware exemplar selection approach for selecting examples
from transfer data sources. Finally, we conduct large-scale experiments on a
variety of complex QA tasks involving numerical reasoning, compositional
complex QA, and heterogeneous complex QA which require decomposed reasoning. We
show that ICAT convincingly outperforms existing prompt-based solutions without
involving any model training, showcasing the benefits of re-using existing
abilities.
Related papers
- AQA: Adaptive Question Answering in a Society of LLMs via Contextual Multi-Armed Bandit [59.10281630985958]
In question answering (QA), different questions can be effectively addressed with different answering strategies.
We develop a dynamic method that adaptively selects the most suitable QA strategy for each question.
Our experiments show that the proposed solution is viable for adaptive orchestration of a QA system with multiple modules.
arXiv Detail & Related papers (2024-09-20T12:28:18Z) - Seek and Solve Reasoning for Table Question Answering [49.006950918895306]
This paper improves Table-based Question Answering (TQA) performance by leveraging Large Language Models' reasoning capabilities.
Inspired by how humans solve TQA tasks, we propose a Seek-and-seek pipeline that instructs the LLM to first seek relevant information and then answer questions.
We present a compact single-stage TQA-solving prompt distilled from the pipeline.
arXiv Detail & Related papers (2024-09-09T02:41:00Z) - DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs [3.24692739098077]
Open-domain complex Question Answering (QA) is a difficult task with challenges in evidence retrieval and reasoning.
We evaluate state-of-the-art pre-trained dense and sparse retrieval models in an open-domain setting.
We observe that late interaction models and surprisingly lexical models like BM25 perform well compared to other pre-trained dense retrieval models.
arXiv Detail & Related papers (2024-06-24T22:09:50Z) - Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity [59.57065228857247]
Retrieval-augmented Large Language Models (LLMs) have emerged as a promising approach to enhancing response accuracy in several tasks, such as Question-Answering (QA)
We propose a novel adaptive QA framework, that can dynamically select the most suitable strategy for (retrieval-augmented) LLMs based on the query complexity.
We validate our model on a set of open-domain QA datasets, covering multiple query complexities, and show that ours enhances the overall efficiency and accuracy of QA systems.
arXiv Detail & Related papers (2024-03-21T13:52:30Z) - ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agent [50.508669199496474]
We develop a ReAct-style LLM agent with the ability to reason and act upon external knowledge.
We refine the agent through a ReST-like method that iteratively trains on previous trajectories.
Starting from a prompted large model and after just two iterations of the algorithm, we can produce a fine-tuned small model.
arXiv Detail & Related papers (2023-12-15T18:20:15Z) - Active Prompting with Chain-of-Thought for Large Language Models [26.5029080638055]
This paper proposes a new method, Active-Prompt, to adapt large language models to different tasks.
By borrowing ideas from the related problem of uncertainty-based active learning, we introduce several metrics to characterize the uncertainty.
Experimental results demonstrate the superiority of our proposed method, achieving state-of-the-art on eight complex reasoning tasks.
arXiv Detail & Related papers (2023-02-23T18:58:59Z) - Successive Prompting for Decomposing Complex Questions [50.00659445976735]
Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting.
We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution.
Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
arXiv Detail & Related papers (2022-12-08T06:03:38Z) - ProQA: Structural Prompt-based Pre-training for Unified Question
Answering [84.59636806421204]
ProQA is a unified QA paradigm that solves various tasks through a single model.
It concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task.
ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios.
arXiv Detail & Related papers (2022-05-09T04:59:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.