A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts

• URL: http://arxiv.org/abs/2101.10196v1
• Date: Mon, 25 Jan 2021 16:01:27 GMT
• Title: A Hybrid Approach to Measure Semantic Relatedness in Biomedical Concepts
• Authors: Katikapalli Subramanyam Kalyan and Sivanesan Sangeetha
• Abstract summary: We generated concept vectors by encoding concept preferred terms using ELMo, BERT, and Sentence BERT models. We trained all the BERT models using Siamese network on SNLI and STSb datasets to allow the models to learn more semantic information. Injecting ontology knowledge into concept vectors further enhances their quality and contributes to better relatedness scores.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Objective: This work aimed to demonstrate the effectiveness of a hybrid approach based on Sentence BERT model and retrofitting algorithm to compute relatedness between any two biomedical concepts. Materials and Methods: We generated concept vectors by encoding concept preferred terms using ELMo, BERT, and Sentence BERT models. We used BioELMo and Clinical ELMo. We used Ontology Knowledge Free (OKF) models like PubMedBERT, BioBERT, BioClinicalBERT, and Ontology Knowledge Injected (OKI) models like SapBERT, CoderBERT, KbBERT, and UmlsBERT. We trained all the BERT models using Siamese network on SNLI and STSb datasets to allow the models to learn more semantic information at the phrase or sentence level so that they can represent multi-word concepts better. Finally, to inject ontology relationship knowledge into concept vectors, we used retrofitting algorithm and concepts from various UMLS relationships. We evaluated our hybrid approach on four publicly available datasets which also includes the recently released EHR-RelB dataset. EHR-RelB is the largest publicly available relatedness dataset in which 89% of terms are multi-word which makes it more challenging. Results: Sentence BERT models mostly outperformed corresponding BERT models. The concept vectors generated using the Sentence BERT model based on SapBERT and retrofitted using UMLS-related concepts achieved the best results on all four datasets. Conclusions: Sentence BERT models are more effective compared to BERT models in computing relatedness scores in most of the cases. Injecting ontology knowledge into concept vectors further enhances their quality and contributes to better relatedness scores.

Related papers

• Multi-level biomedical NER through multi-granularity embeddings and enhanced labeling [3.8599767910528917]
  This paper proposes a hybrid approach that integrates the strengths of multiple models. BERT provides contextualized word embeddings, a pre-trained multi-channel CNN for character-level information capture, and following by a BiLSTM + CRF for sequence labelling and modelling dependencies between the words in the text. We evaluate our model on the benchmark i2b2/2010 dataset, achieving an F1-score of 90.11.
  arXiv  Detail & Related papers  (2023-12-24T21:45:36Z)
• Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs [54.223394825528665]
  We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models. We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
  arXiv  Detail & Related papers  (2023-12-21T14:26:57Z)
• Evaluating Biomedical BERT Models for Vocabulary Alignment at Scale in the UMLS Metathesaurus [8.961270657070942]
  The current UMLS (Unified Medical Language System) Metathesaurus construction process is expensive and error-prone. Recent advances in Natural Language Processing have achieved state-of-the-art (SOTA) performance on downstream tasks. We aim to validate if approaches using the BERT models can actually outperform the existing approaches for predicting synonymy in the UMLS Metathesaurus.
  arXiv  Detail & Related papers  (2021-09-14T16:52:16Z)
• Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT [17.739843061394367]
  Mixture-of-Partitions (MoP) can handle a very large knowledge graph (KG) by partitioning it into smaller sub-graphs and infusing their specific knowledge into various BERT models using lightweight adapters. We evaluate our MoP with three biomedical BERTs (SciBERT, BioBERT, PubmedBERT) on six downstream tasks (inc. NLI, QA, Classification), and the results show that our MoP consistently enhances the underlying BERTs in task performance.
  arXiv  Detail & Related papers  (2021-09-10T11:54:25Z)
• Scientific Language Models for Biomedical Knowledge Base Completion: An Empirical Study [62.376800537374024]
  We study scientific LMs for KG completion, exploring whether we can tap into their latent knowledge to enhance biomedical link prediction. We integrate the LM-based models with KG embedding models, using a router method that learns to assign each input example to either type of model and provides a substantial boost in performance.
  arXiv  Detail & Related papers  (2021-06-17T17:55:33Z)
• Fast and Effective Biomedical Entity Linking Using a Dual Encoder [48.86736921025866]
  We propose a BERT-based dual encoder model that resolves multiple mentions in a document in one shot. We show that our proposed model is multiple times faster than existing BERT-based models while being competitive in accuracy for biomedical entity linking.
  arXiv  Detail & Related papers  (2021-03-08T19:32:28Z)
• Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks [4.955649816620742]
  This paper explores on sentence embedding models for BERT and ALBERT. We take a modified BERT network with siamese and triplet network structures called Sentence-BERT (SBERT) and replace BERT with ALBERT to create Sentence-ALBERT (SALBERT)
  arXiv  Detail & Related papers  (2021-01-26T09:14:06Z)
• Investigation of BERT Model on Biomedical Relation Extraction Based on Revised Fine-tuning Mechanism [2.8881198461098894]
  We will investigate the method of utilizing the entire layer in the fine-tuning process of BERT model. In addition, further analysis indicates that the key knowledge about the relations can be learned from the last layer of BERT model.
  arXiv  Detail & Related papers  (2020-11-01T01:47:16Z)
• UmlsBERT: Clinical Domain Knowledge Augmentation of Contextual Embeddings Using the Unified Medical Language System Metathesaurus [73.86656026386038]
  We introduce UmlsBERT, a contextual embedding model that integrates domain knowledge during the pre-training process. By applying these two strategies, UmlsBERT can encode clinical domain knowledge into word embeddings and outperform existing domain-specific models.
  arXiv  Detail & Related papers  (2020-10-20T15:56:31Z)
• Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks [62.9447303059342]
  We show the importance of this problem in medical community. We present a modification of Bidirectional Representations from Transformers (BERT) model for classification sequence. We use a large-scale Russian EHR dataset consisting of about 4 million unique patient visits.
  arXiv  Detail & Related papers  (2020-07-15T09:22:55Z)
• Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage [65.7062363323781]
  We propose a novel framework based on BioBERT (Bidirectional Representations from Transformers forBiomedical TextMining) We introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient portal message triage that classifies the urgency of a message into three categories: non-urgent, medium and urgent.
  arXiv  Detail & Related papers  (2020-06-22T03:39:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.