Unsupervised Pre-training for Biomedical Question Answering

• URL: http://arxiv.org/abs/2009.12952v1
• Date: Sun, 27 Sep 2020 21:07:51 GMT
• Title: Unsupervised Pre-training for Biomedical Question Answering
• Authors: Vaishnavi Kommaraju, Karthick Gunasekaran, Kun Li, Trapit Bansal, Andrew McCallum, Ivana Williams, Ana-Maria Istrate
• Abstract summary: We introduce a new pre-training task from unlabeled data designed to reason about biomedical entities in the context. Our experiments show that pre-training BioBERT on the proposed pre-training task significantly boosts performance and outperforms the previous best model from the 7th BioASQ Task 7b-Phase B challenge.
• Score: 32.525495687236194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We explore the suitability of unsupervised representation learning methods on biomedical text -- BioBERT, SciBERT, and BioSentVec -- for biomedical question answering. To further improve unsupervised representations for biomedical QA, we introduce a new pre-training task from unlabeled data designed to reason about biomedical entities in the context. Our pre-training method consists of corrupting a given context by randomly replacing some mention of a biomedical entity with a random entity mention and then querying the model with the correct entity mention in order to locate the corrupted part of the context. This de-noising task enables the model to learn good representations from abundant, unlabeled biomedical text that helps QA tasks and minimizes the train-test mismatch between the pre-training task and the downstream QA tasks by requiring the model to predict spans. Our experiments show that pre-training BioBERT on the proposed pre-training task significantly boosts performance and outperforms the previous best model from the 7th BioASQ Task 7b-Phase B challenge.

Related papers

• BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments [112.25067497985447]
  We introduce BioDiscoveryAgent, an agent that designs new experiments, reasons about their outcomes, and efficiently navigates the hypothesis space to reach desired solutions. BioDiscoveryAgent can uniquely design new experiments without the need to train a machine learning model. It achieves an average of 21% improvement in predicting relevant genetic perturbations across six datasets.
  arXiv  Detail & Related papers  (2024-05-27T19:57:17Z)
• BMRetriever: Tuning Large Language Models as Better Biomedical Text Retrievers [48.21255861863282]
  BMRetriever is a series of dense retrievers for enhancing biomedical retrieval. BMRetriever exhibits strong parameter efficiency, with the 410M variant outperforming baselines up to 11.7 times larger.
  arXiv  Detail & Related papers  (2024-04-29T05:40:08Z)
• An Evaluation of Large Language Models in Bioinformatics Research [52.100233156012756]
  We study the performance of large language models (LLMs) on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks.
  arXiv  Detail & Related papers  (2024-02-21T11:27:31Z)
• BIOptimus: Pre-training an Optimal Biomedical Language Model with Curriculum Learning for Named Entity Recognition [0.0]
  Using language models (LMs) pre-trained in a self-supervised setting on large corpora has helped to deal with the problem of limited label data. Recent research in biomedical language processing has offered a number of biomedical LMs pre-trained. This paper aims to investigate different pre-training methods, such as pre-training the biomedical LM from scratch and pre-training it in a continued fashion.
  arXiv  Detail & Related papers  (2023-08-16T18:48:01Z)
• Enhancing Biomedical Text Summarization and Question-Answering: On the Utility of Domain-Specific Pre-Training [10.267057557137665]
  We identify a suitable model architecture and use it to show a benefit of a general-domain pre-training followed by a task-specific fine-tuning. Our results indicate that a Large Language Model without domain-specific pre-training can have a significant edge in some domain-specific biomedical text generation tasks.
  arXiv  Detail & Related papers  (2023-07-10T08:32:45Z)
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)
• Slot Filling for Biomedical Information Extraction [0.5330240017302619]
  We present a slot filling approach to the task of biomedical IE. We follow the proposed paradigm of coupling a Tranformer-based bi-encoder, Dense Passage Retrieval, with a Transformer-based reader model.
  arXiv  Detail & Related papers  (2021-09-17T14:16:00Z)
• An Experimental Evaluation of Transformer-based Language Models in the Biomedical Domain [0.984441002699829]
  This paper summarizes experiments conducted in replicating BioBERT and further pre-training and fine-tuning in the biomedical domain. We also investigate the effectiveness of domain-specific and domain-agnostic pre-trained models across downstream biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-12-31T03:09:38Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)
• Transferability of Natural Language Inference to Biomedical Question Answering [17.38537039378825]
  We focus on applying BioBERT to transfer the knowledge of natural language inference (NLI) to biomedical question answering (QA) We observe that BioBERT trained on the NLI dataset obtains better performance on Yes/No (+5.59%), Factoid (+0.53%), List type (+13.58%) questions. We present a sequential transfer learning method that significantly performed well in the 8th BioASQ Challenge (Phase B)
  arXiv  Detail & Related papers  (2020-07-01T04:05:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.