Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies
- URL: http://arxiv.org/abs/2102.08366v1
- Date: Tue, 16 Feb 2021 18:51:13 GMT
- Title: Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies
- Authors: Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, Yulan He
- Abstract summary: Biomedical question-answering (QA) has gained increased attention for its capability to provide users with high-quality information from a vast scientific literature.
We propose a simple yet unexplored approach, which we call biomedical entity-aware masking (BEM)
We encourage masked language models to learn entity-centric knowledge based on the pivotal entities characterizing the domain at hand, and employ those entities to drive the LM fine-tuning. Experimental results show performance on par with state-of-the-art models on several biomedical QA datasets.
- Score: 25.990479833023166
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Biomedical question-answering (QA) has gained increased attention for its
capability to provide users with high-quality information from a vast
scientific literature. Although an increasing number of biomedical QA datasets
has been recently made available, those resources are still rather limited and
expensive to produce. Transfer learning via pre-trained language models (LMs)
has been shown as a promising approach to leverage existing general-purpose
knowledge. However, finetuning these large models can be costly and time
consuming, often yielding limited benefits when adapting to specific themes of
specialised domains, such as the COVID-19 literature. To bootstrap further
their domain adaptation, we propose a simple yet unexplored approach, which we
call biomedical entity-aware masking (BEM). We encourage masked language models
to learn entity-centric knowledge based on the pivotal entities characterizing
the domain at hand, and employ those entities to drive the LM fine-tuning. The
resulting strategy is a downstream process applicable to a wide variety of
masked LMs, not requiring additional memory or components in the neural
architectures. Experimental results show performance on par with
state-of-the-art models on several biomedical QA datasets.
Related papers
- Adapter-based Approaches to Knowledge-enhanced Language Models -- A Survey [48.52320309766703]
Knowledge-enhanced language models (KELMs) have emerged as promising tools to bridge the gap between large-scale language models and domain-specific knowledge.
KELMs can achieve higher factual accuracy and hallucinations by leveraging knowledge graphs (KGs)
arXiv Detail & Related papers (2024-11-25T14:10:24Z) - NeuroSym-BioCAT: Leveraging Neuro-Symbolic Methods for Biomedical Scholarly Document Categorization and Question Answering [0.14999444543328289]
We introduce a novel approach that integrates an optimized topic modelling framework, OVB-LDA, with the BI-POP CMA-ES optimization technique for enhanced scholarly document abstract categorization.
We employ the distilled MiniLM model, fine-tuned on domain-specific data, for high-precision answer extraction.
arXiv Detail & Related papers (2024-10-29T14:45:12Z) - BioMNER: A Dataset for Biomedical Method Entity Recognition [25.403593761614424]
We propose a novel dataset for biomedical method entity recognition.
We employ an automated BioMethod entity recognition and information retrieval system to assist human annotation.
Our empirical findings reveal that the large parameter counts of language models surprisingly inhibit the effective assimilation of entity extraction patterns.
arXiv Detail & Related papers (2024-06-28T16:34:24Z) - UltraMedical: Building Specialized Generalists in Biomedicine [40.53028639007486]
We present the UltraMedical collections, which consist of high-quality manual and synthetic datasets in the biomedicine domain.
We fine-tune a suite of specialized medical models based on Llama-3 series, demonstrating breathtaking capabilities across various medical benchmarks.
arXiv Detail & Related papers (2024-06-06T10:50:26Z) - Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed.
In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset.
We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
arXiv Detail & Related papers (2024-04-27T05:03:42Z) - Diversifying Knowledge Enhancement of Biomedical Language Models using
Adapter Modules and Knowledge Graphs [54.223394825528665]
We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models.
We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT.
We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
arXiv Detail & Related papers (2023-12-21T14:26:57Z) - Interpretability from a new lens: Integrating Stratification and Domain
knowledge for Biomedical Applications [0.0]
This paper proposes a novel computational strategy for the stratification of biomedical problem datasets into k-fold cross-validation (CVs)
This approach can improve model stability, establish trust, and provide explanations for outcomes generated by trained IML models.
arXiv Detail & Related papers (2023-03-15T12:02:02Z) - Fine-Tuning Large Neural Language Models for Biomedical Natural Language
Processing [55.52858954615655]
We conduct a systematic study on fine-tuning stability in biomedical NLP.
We show that finetuning performance may be sensitive to pretraining settings, especially in low-resource domains.
We show that these techniques can substantially improve fine-tuning performance for lowresource biomedical NLP applications.
arXiv Detail & Related papers (2021-12-15T04:20:35Z) - Scientific Language Models for Biomedical Knowledge Base Completion: An
Empirical Study [62.376800537374024]
We study scientific LMs for KG completion, exploring whether we can tap into their latent knowledge to enhance biomedical link prediction.
We integrate the LM-based models with KG embedding models, using a router method that learns to assign each input example to either type of model and provides a substantial boost in performance.
arXiv Detail & Related papers (2021-06-17T17:55:33Z) - Machine Learning in Nano-Scale Biomedical Engineering [77.75587007080894]
We review the existing research regarding the use of machine learning in nano-scale biomedical engineering.
The main challenges that can be formulated as ML problems are classified into the three main categories.
For each of the presented methodologies, special emphasis is given to its principles, applications, and limitations.
arXiv Detail & Related papers (2020-08-05T15:45:54Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.