Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs
Answering
- URL: http://arxiv.org/abs/2304.13911v2
- Date: Fri, 30 Jun 2023 13:21:36 GMT
- Title: Federated Prompting and Chain-of-Thought Reasoning for Improving LLMs
Answering
- Authors: Xiangyang Liu, Tianqi Pang, Chenyou Fan
- Abstract summary: We investigate how to enhance answer precision in frequently asked questions posed by distributed users using cloud-based Large Language Models (LLMs)
Our study focuses on a typical situations where users ask similar queries that involve identical mathematical reasoning steps and problem-solving procedures.
We propose to improve the distributed synonymous questions using Self-Consistency (SC) and Chain-of-Thought (CoT) techniques.
- Score: 13.735277588793997
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We investigate how to enhance answer precision in frequently asked questions
posed by distributed users using cloud-based Large Language Models (LLMs). Our
study focuses on a typical situations where users ask similar queries that
involve identical mathematical reasoning steps and problem-solving procedures.
Due to the unsatisfactory accuracy of LLMs' zero-shot prompting with standalone
questions, we propose to improve the distributed synonymous questions using
Self-Consistency (SC) and Chain-of-Thought (CoT) techniques. Specifically, we
first retrieve synonymous questions from a crowd-sourced database and create a
federated question pool. We call these federated synonymous questions with the
same or different parameters SP-questions or DP-questions, respectively. We
refer to our methods as Fed-SP-SC and Fed-DP-CoT, which can generate
significantly more accurate answers for all user queries without requiring
sophisticated model-tuning. Through extensive experiments, we demonstrate that
our proposed methods can significantly enhance question accuracy by fully
exploring the synonymous nature of the questions and the consistency of the
answers.
Related papers
- CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval [52.134133938779776]
We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate.
Our approach works by augmenting a large language model (LLM) to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn.
arXiv Detail & Related papers (2024-04-28T18:21:31Z) - Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based
Question Answering [62.14682452663157]
We propose a novel Chain-of-Discussion framework to leverage the synergy among open-source Large Language Models.
Our experiments show that discussions among multiple LLMs play a vital role in enhancing the quality of answers.
arXiv Detail & Related papers (2024-02-26T05:31:34Z) - Enhancing Answer Selection in Community Question Answering with
Pre-trained and Large Language Models [0.9065034043031668]
We first propose the Question-Answer cross attention networks (QAN) with pre-trained models for answer selection.
We then utilize large language model (LLM) to perform answer selection with knowledge augmentation.
Experiments show that the QAN model state-of-the-art performance on two datasets, SemEval2015 and SemEval 2017.
arXiv Detail & Related papers (2023-11-29T10:24:50Z) - Improving Zero-shot Visual Question Answering via Large Language Models
with Reasoning Question Prompts [22.669502403623166]
We present Reasoning Question Prompts for VQA tasks, which can further activate the potential of Large Language Models.
We generate self-contained questions as reasoning question prompts via an unsupervised question edition module.
Each reasoning question prompt clearly indicates the intent of the original question.
Then, the candidate answers associated with their confidence scores acting as answer integritys are fed into LLMs.
arXiv Detail & Related papers (2023-11-15T15:40:46Z) - Multi-Tenant Optimization For Few-Shot Task-Oriented FAQ Retrieval [0.0]
Business-specific Frequently Asked Questions (FAQ) retrieval in task-oriented dialog systems poses unique challenges.
We evaluate performance for such Business FAQ using query-Question (q-Q) similarity and few-shot intent detection techniques.
We propose a novel approach to scale multi-tenant FAQ applications in real-world context by contrastive fine-tuning of the last layer in sentence Bi-Encoders along with tenant-specific weight switching.
arXiv Detail & Related papers (2023-01-25T10:55:45Z) - Momentum Contrastive Pre-training for Question Answering [54.57078061878619]
MCROSS introduces a momentum contrastive learning framework to align the answer probability between cloze-like and natural query-passage sample pairs.
Our method achieves noticeable improvement compared with all baselines in both supervised and zero-shot scenarios.
arXiv Detail & Related papers (2022-12-12T08:28:22Z) - Successive Prompting for Decomposing Complex Questions [50.00659445976735]
Recent works leverage the capabilities of large language models (LMs) to perform complex question answering in a few-shot setting.
We introduce Successive Prompting'', where we iteratively break down a complex task into a simple task, solve it, and then repeat the process until we get the final solution.
Our best model (with successive prompting) achieves an improvement of 5% absolute F1 on a few-shot version of the DROP dataset.
arXiv Detail & Related papers (2022-12-08T06:03:38Z) - CREPE: Open-Domain Question Answering with False Presuppositions [92.20501870319765]
We introduce CREPE, a QA dataset containing a natural distribution of presupposition failures from online information-seeking forums.
We find that 25% of questions contain false presuppositions, and provide annotations for these presuppositions and their corrections.
We show that adaptations of existing open-domain QA models can find presuppositions moderately well, but struggle when predicting whether a presupposition is factually correct.
arXiv Detail & Related papers (2022-11-30T18:54:49Z) - Diverse Multi-Answer Retrieval with Determinantal Point Processes [11.925050407713597]
We propose a re-ranking based approach using Determinantal point processes utilizing BERT as kernels.
Results demonstrate that our re-ranking technique outperforms state-of-the-art method on the AmbigQA dataset.
arXiv Detail & Related papers (2022-11-29T08:54:05Z) - A Mutual Information Maximization Approach for the Spurious Solution
Problem in Weakly Supervised Question Answering [60.768146126094955]
Weakly supervised question answering usually has only the final answers as supervision signals.
There may exist many spurious solutions that coincidentally derive the correct answer, but training on such solutions can hurt model performance.
We propose to explicitly exploit such semantic correlations by maximizing the mutual information between question-answer pairs and predicted solutions.
arXiv Detail & Related papers (2021-06-14T05:47:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.