A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19

• URL: http://arxiv.org/abs/2006.10964v2
• Date: Tue, 23 Jun 2020 20:23:04 GMT
• Title: A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19
• Authors: David Oniani, Yanshan Wang
• Abstract summary: COVID-19 has caused more than 7.4 million cases and over 418,000 deaths. Online communities, forums, and social media provide potential venues to search for relevant questions and answers. We propose to apply a language model for automatically answering questions related to COVID-19 and qualitatively evaluate the generated responses.
• Score: 4.676651062800037
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: COVID-19 has resulted in an ongoing pandemic and as of 12 June 2020, has caused more than 7.4 million cases and over 418,000 deaths. The highly dynamic and rapidly evolving situation with COVID-19 has made it difficult to access accurate, on-demand information regarding the disease. Online communities, forums, and social media provide potential venues to search for relevant questions and answers, or post questions and seek answers from other members. However, due to the nature of such sites, there are always a limited number of relevant questions and responses to search from, and posted questions are rarely answered immediately. With the advancements in the field of natural language processing, particularly in the domain of language models, it has become possible to design chatbots that can automatically answer consumer questions. However, such models are rarely applied and evaluated in the healthcare domain, to meet the information needs with accurate and up-to-date healthcare data. In this paper, we propose to apply a language model for automatically answering questions related to COVID-19 and qualitatively evaluate the generated responses. We utilized the GPT-2 language model and applied transfer learning to retrain it on the COVID-19 Open Research Dataset (CORD-19) corpus. In order to improve the quality of the generated responses, we applied 4 different approaches, namely tf-idf, BERT, BioBERT, and USE to filter and retain relevant sentences in the responses. In the performance evaluation step, we asked two medical experts to rate the responses. We found that BERT and BioBERT, on average, outperform both tf-idf and USE in relevance-based sentence filtering tasks. Additionally, based on the chatbot, we created a user-friendly interactive web application to be hosted online.

Related papers

• Efficient Medical Question Answering with Knowledge-Augmented Question Generation [5.145812785735094]
  We introduce a method to improve the proficiency of a small language model in the medical domain by employing a two-fold approach. We first fine-tune the model on a corpus of medical textbooks. Then, we use GPT-4 to generate questions similar to the downstream task, prompted with textbook knowledge, and use them to fine-tune the model.
  arXiv  Detail & Related papers  (2024-05-23T14:53:52Z)
• CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrieval [52.134133938779776]
  We present CLARINET, a system that asks informative clarification questions by choosing questions whose answers would maximize certainty in the correct candidate. Our approach works by augmenting a large language model (LLM) to condition on a retrieval distribution, finetuning end-to-end to generate the question that would have maximized the rank of the true candidate at each turn.
  arXiv  Detail & Related papers  (2024-04-28T18:21:31Z)
• Using Weak Supervision and Data Augmentation in Question Answering [0.12499537119440242]
  The onset of the COVID-19 pandemic accentuated the need for access to biomedical literature to answer timely and disease-specific questions. We explore the roles weak supervision and data augmentation play in training deep neural network QA models. We evaluate our methods in the context of QA models at the core of a system to answer questions about COVID-19.
  arXiv  Detail & Related papers  (2023-09-28T05:16:51Z)
• ExpertQA: Expert-Curated Questions and Attributed Answers [51.68314045809179]
  We conduct human evaluation of responses from a few representative systems along various axes of attribution and factuality. We collect expert-curated questions from 484 participants across 32 fields of study, and then ask the same experts to evaluate generated responses to their own questions. The output of our analysis is ExpertQA, a high-quality long-form QA dataset with 2177 questions spanning 32 fields, along with verified answers and attributions for claims in the answers.
  arXiv  Detail & Related papers  (2023-09-14T16:54:34Z)
• Top K Relevant Passage Retrieval for Biomedical Question Answering [1.0636004442689055]
  Question answering is a task that answers factoid questions using a large collection of documents. The existing Dense Passage Retrieval model has been trained on Wikipedia dump from Dec. 20, 2018, as the source documents for answering questions. In this work, we work on the existing DPR framework for the biomedical domain and retrieve answers from the Pubmed articles which is a reliable source to answer medical questions.
  arXiv  Detail & Related papers  (2023-08-08T04:06:11Z)
• WebCPM: Interactive Web Search for Chinese Long-form Question Answering [104.676752359777]
  Long-form question answering (LFQA) aims at answering complex, open-ended questions with detailed, paragraph-length responses. We introduce WebCPM, the first Chinese LFQA dataset. We collect 5,500 high-quality question-answer pairs, together with 14,315 supporting facts and 121,330 web search actions.
  arXiv  Detail & Related papers  (2023-05-11T14:47:29Z)
• Medical Question Understanding and Answering with Knowledge Grounding and Semantic Self-Supervision [53.692793122749414]
  We introduce a medical question understanding and answering system with knowledge grounding and semantic self-supervision. Our system is a pipeline that first summarizes a long, medical, user-written question, using a supervised summarization loss. The system first matches the summarized user question with an FAQ from a trusted medical knowledge base, and then retrieves a fixed number of relevant sentences from the corresponding answer document.
  arXiv  Detail & Related papers  (2022-09-30T08:20:32Z)
• UIT-ViCoV19QA: A Dataset for COVID-19 Community-based Question Answering on Vietnamese Language [0.0]
  We present the first Vietnamese community-based question answering dataset for developing question answering systems for COVID-19 called UIT-ViCoV19QA. The dataset comprises 4,500 question-answer pairs collected from trusted medical sources, with at least one answer and at most four unique paraphrased answers per question.
  arXiv  Detail & Related papers  (2022-09-14T14:24:23Z)
• Inquisitive Question Generation for High Level Text Comprehension [60.21497846332531]
  We introduce INQUISITIVE, a dataset of 19K questions that are elicited while a person is reading through a document. We show that readers engage in a series of pragmatic strategies to seek information. We evaluate question generation models based on GPT-2 and show that our model is able to generate reasonable questions.
  arXiv  Detail & Related papers  (2020-10-04T19:03:39Z)
• Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQs [5.512295869673147]
  We show how a double fine-tuning approach of pretraining a neural network on medical question-answer pairs is a useful intermediate task for determining medical question similarity. We also describe a currently live system that uses the trained model to match user questions to COVID-related FAQ.
  arXiv  Detail & Related papers  (2020-08-04T18:20:04Z)
• On the Generation of Medical Dialogues for COVID-19 [60.63485429268256]
  People experiencing COVID19-related symptoms or exposed to risk factors have a pressing need to consult doctors. Because of the shortage of medical professionals, many people cannot receive online consultations timely. We aim to develop a medical dialogue system that can provide COVID19-related consultations.
  arXiv  Detail & Related papers  (2020-05-11T21:23:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.