Mixture of Experts for Biomedical Question Answering

• URL: http://arxiv.org/abs/2204.07469v1
• Date: Fri, 15 Apr 2022 14:11:40 GMT
• Title: Mixture of Experts for Biomedical Question Answering
• Authors: Damai Dai, Wenbin Jiang, Jiyuan Zhang, Weihua Peng, Yajuan Lyu, Zhifang Sui, Baobao Chang, Yong Zhu
• Abstract summary: We propose a Mixture-of-Expert (MoE) based question answering method called MoEBQA. MoEBQA decouples the computation for different types of questions by sparse routing. We evaluate MoEBQA on three Biomedical Question Answering (BQA) datasets constructed based on real examinations.
• Score: 34.92691831878302
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Biomedical Question Answering (BQA) has attracted increasing attention in recent years due to its promising application prospect. It is a challenging task because the biomedical questions are professional and usually vary widely. Existing question answering methods answer all questions with a homogeneous model, leading to various types of questions competing for the shared parameters, which will confuse the model decision for each single type of questions. In this paper, in order to alleviate the parameter competition problem, we propose a Mixture-of-Expert (MoE) based question answering method called MoEBQA that decouples the computation for different types of questions by sparse routing. To be specific, we split a pretrained Transformer model into bottom and top blocks. The bottom blocks are shared by all the examples, aiming to capture the general features. The top blocks are extended to an MoE version that consists of a series of independent experts, where each example is assigned to a few experts according to its underlying question type. MoEBQA automatically learns the routing strategy in an end-to-end manner so that each expert tends to deal with the question types it is expert in. We evaluate MoEBQA on three BQA datasets constructed based on real examinations. The results show that our MoE extension significantly boosts the performance of question answering models and achieves new state-of-the-art performance. In addition, we elaborately analyze our MoE modules to reveal how MoEBQA works and find that it can automatically group the questions into human-readable clusters.

Related papers

• Multi-LLM QA with Embodied Exploration [55.581423861790945]
  We investigate the use of Multi-Embodied LLM Explorers (MELE) for question-answering in an unknown environment. Multiple LLM-based agents independently explore and then answer queries about a household environment. We analyze different aggregation methods to generate a single, final answer for each query.
  arXiv  Detail & Related papers  (2024-06-16T12:46:40Z)
• Unsupervised multiple choices question answering via universal corpus [27.78825771434918]
  We propose a novel framework designed to generate synthetic multiple choice question answering (MCQA) data. We leverage both named entities (NE) and knowledge graphs to discover plausible distractors to form complete synthetic samples.
  arXiv  Detail & Related papers  (2024-02-27T09:10:28Z)
• An Empirical Comparison of LM-based Question and Answer Generation Methods [79.31199020420827]
  Question and answer generation (QAG) consists of generating a set of question-answer pairs given a context. In this paper, we establish baselines with three different QAG methodologies that leverage sequence-to-sequence language model (LM) fine-tuning. Experiments show that an end-to-end QAG model, which is computationally light at both training and inference times, is generally robust and outperforms other more convoluted approaches.
  arXiv  Detail & Related papers  (2023-05-26T14:59:53Z)
• Getting MoRE out of Mixture of Language Model Reasoning Experts [71.61176122960464]
  We propose a Mixture-of-Reasoning-Experts (MoRE) framework that ensembles diverse specialized language models. We specialize the backbone language model with prompts optimized for different reasoning categories, including factual, multihop, mathematical, and commonsense reasoning. Our human study confirms that presenting expert predictions and the answer selection process helps annotators more accurately calibrate when to trust the system's output.
  arXiv  Detail & Related papers  (2023-05-24T02:00:51Z)
• MetaQA: Combining Expert Agents for Multi-Skill Question Answering [49.35261724460689]
  We argue that despite the promising results of multi-dataset models, some domains or QA formats might require specific architectures. We propose to combine expert agents with a novel, flexible, and training-efficient architecture that considers questions, answer predictions, and answer-prediction confidence scores.
  arXiv  Detail & Related papers  (2021-12-03T14:05:52Z)
• Challenges in Generalization in Open Domain Question Answering [16.63912089965166]
  We introduce and annotate questions according to three categories that measure different levels and kinds of generalization. Key question difficulty factors are cascading errors from the retrieval component, frequency of question pattern, and frequency of the entity.
  arXiv  Detail & Related papers  (2021-09-02T18:04:10Z)
• Sequence Tagging for Biomedical Extractive Question Answering [12.464143741310137]
  We investigate the difference of the question distribution across the general and biomedical domains. We discover biomedical questions are more likely to require list-type answers (multiple answers) than factoid-type answers (single answer) Our approach can learn to decide the number of answers for a question from training data.
  arXiv  Detail & Related papers  (2021-04-15T15:42:34Z)
• Unsupervised Question Decomposition for Question Answering [102.56966847404287]
  We propose an algorithm for One-to-N Unsupervised Sequence Sequence (ONUS) that learns to map one hard, multi-hop question to many simpler, single-hop sub-questions. We show large QA improvements on HotpotQA over a strong baseline on the original, out-of-domain, and multi-hop dev sets.
  arXiv  Detail & Related papers  (2020-02-22T19:40:35Z)
• ManyModalQA: Modality Disambiguation and QA over Diverse Inputs [73.93607719921945]
  We present a new multimodal question answering challenge, ManyModalQA, in which an agent must answer a question by considering three distinct modalities. We collect our data by scraping Wikipedia and then utilize crowdsourcing to collect question-answer pairs.
  arXiv  Detail & Related papers  (2020-01-22T14:39:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.