CLAM: Selective Clarification for Ambiguous Questions with Large Language Models

• URL: http://arxiv.org/abs/2212.07769v1
• Date: Thu, 15 Dec 2022 12:47:18 GMT
• Title: CLAM: Selective Clarification for Ambiguous Questions with Large Language Models
• Authors: Lorenz Kuhn, Yarin Gal, Sebastian Farquhar
• Abstract summary: We show that current SotA models do not ask the user for clarification when presented with imprecise questions. We introduce CLAM, a framework that first uses the model to detect ambiguous questions and if an ambiguous question is detected, prompts the model to ask the user for clarification. We show that our method achieves a 20.15 percentage point accuracy improvement over SotA on a novel ambiguous question-answering answering data set.
• Score: 37.37606905433334
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: State-of-the-art language models are often accurate on many question-answering benchmarks with well-defined questions. Yet, in real settings questions are often unanswerable without asking the user for clarifying information. We show that current SotA models often do not ask the user for clarification when presented with imprecise questions and instead provide incorrect answers or "hallucinate". To address this, we introduce CLAM, a framework that first uses the model to detect ambiguous questions, and if an ambiguous question is detected, prompts the model to ask the user for clarification. Furthermore, we show how to construct a scalable and cost-effective automatic evaluation protocol using an oracle language model with privileged information to provide clarifying information. We show that our method achieves a 20.15 percentage point accuracy improvement over SotA on a novel ambiguous question-answering answering data set derived from TriviaQA.

Related papers

• Localizing and Mitigating Errors in Long-form Question Answering [79.63372684264921]
  Long-form question answering (LFQA) aims to provide thorough and in-depth answers to complex questions, enhancing comprehension. This work introduces HaluQuestQA, the first hallucination dataset with localized error annotations for human-written and model-generated LFQA answers.
  arXiv  Detail & Related papers  (2024-07-16T17:23:16Z)
• CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval [52.134133938779776]
  We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate. Our approach works by augmenting a large language model (LLM) to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn.
  arXiv  Detail & Related papers  (2024-04-28T18:21:31Z)
• Don't Just Say "I don't know"! Self-aligning Large Language Models for Responding to Unknown Questions with Explanations [70.6395572287422]
  Self-alignment method is capable of not only refusing to answer but also providing explanation to the unanswerability of unknown questions. We conduct disparity-driven self-curation to select qualified data for fine-tuning the LLM itself for aligning the responses to unknown questions as desired.
  arXiv  Detail & Related papers  (2024-02-23T02:24:36Z)
• Uncertainty-aware Language Modeling for Selective Question Answering [107.47864420630923]
  We present an automatic large language model (LLM) conversion approach that produces uncertainty-aware LLMs. Our approach is model- and data-agnostic, is computationally-efficient, and does not rely on external models or systems.
  arXiv  Detail & Related papers  (2023-11-26T22:47:54Z)
• Selectively Answering Ambiguous Questions [38.83930394700588]
  We find that the most reliable approach to decide when to abstain involves quantifying repetition within sampled model outputs. Our results suggest that sampling-based confidence scores help calibrate answers to relatively unambiguous questions.
  arXiv  Detail & Related papers  (2023-05-24T01:25:38Z)
• Realistic Conversational Question Answering with Answer Selection based on Calibrated Confidence and Uncertainty Measurement [54.55643652781891]
  Conversational Question Answering (ConvQA) models aim at answering a question with its relevant paragraph and previous question-answer pairs that occurred during conversation multiple times. We propose to filter out inaccurate answers in the conversation history based on their estimated confidences and uncertainties from the ConvQA model. We validate our models, Answer Selection-based realistic Conversation Question Answering, on two standard ConvQA datasets.
  arXiv  Detail & Related papers  (2023-02-10T09:42:07Z)
• Answering Ambiguous Questions through Generative Evidence Fusion and Round-Trip Prediction [46.38201136570501]
  We present a model that aggregates and combines evidence from multiple passages to adaptively predict a single answer or a set of question-answer pairs for ambiguous questions. Our model, named Refuel, achieves a new state-of-the-art performance on the AmbigQA dataset, and shows competitive performance on NQ-Open and TriviaQA.
  arXiv  Detail & Related papers  (2020-11-26T05:48:55Z)
• A Wrong Answer or a Wrong Question? An Intricate Relationship between Question Reformulation and Answer Selection in Conversational Question Answering [15.355557454305776]
  We show that question rewriting (QR) of the conversational context allows to shed more light on this phenomenon. We present the results of this analysis on the TREC CAsT and QuAC (CANARD) datasets.
  arXiv  Detail & Related papers  (2020-10-13T06:29:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.