Automated Text Mining of Experimental Methodologies from Biomedical Literature

• URL: http://arxiv.org/abs/2404.13779v1
• Date: Sun, 21 Apr 2024 21:19:36 GMT
• Title: Automated Text Mining of Experimental Methodologies from Biomedical Literature
• Authors: Ziqing Guo,
• Abstract summary: DistilBERT is a methodology-specific, pre-trained generative classification language model for mining biomedicine texts. It has proven its effectiveness in linguistic understanding capabilities and has reduced the size of BERT models by 40% but by 60% faster. Our aim is to integrate this highly specialised and specific model into different research industries.
• Score: 0.087024326813104
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Biomedical literature is a rapidly expanding field of science and technology. Classification of biomedical texts is an essential part of biomedicine research, especially in the field of biology. This work proposes the fine-tuned DistilBERT, a methodology-specific, pre-trained generative classification language model for mining biomedicine texts. The model has proven its effectiveness in linguistic understanding capabilities and has reduced the size of BERT models by 40\% but by 60\% faster. The main objective of this project is to improve the model and assess the performance of the model compared to the non-fine-tuned model. We used DistilBert as a support model and pre-trained on a corpus of 32,000 abstracts and complete text articles; our results were impressive and surpassed those of traditional literature classification methods by using RNN or LSTM. Our aim is to integrate this highly specialised and specific model into different research industries.

Related papers

• Pre-training data selection for biomedical domain adaptation using journal impact metrics [0.0]
  We employ two straightforward journal impact metrics and conduct experiments by continually pre-training BERT on various subsets of the complete PubMed training set. Our results show that pruning using journal impact metrics is not efficient. But we also show that pre-training using fewer abstracts (but with the same number of training steps) does not necessarily decrease the resulting model's performance.
  arXiv  Detail & Related papers  (2024-09-04T13:59:48Z)
• BioMamba: A Pre-trained Biomedical Language Representation Model Leveraging Mamba [11.012644129839922]
  We present BioMamba, a pre-trained model specifically designed for biomedical text mining. Our empirical studies demonstrate that BioMamba significantly outperforms models like BioBERT and general-domain Mamba across various biomedical tasks. For instance, BioMamba achieves a 100 times reduction in perplexity and a 4 times reduction in cross-entropy loss on the BioASQ test set.
  arXiv  Detail & Related papers  (2024-08-05T16:21:36Z)
• Multi-level biomedical NER through multi-granularity embeddings and enhanced labeling [3.8599767910528917]
  This paper proposes a hybrid approach that integrates the strengths of multiple models. BERT provides contextualized word embeddings, a pre-trained multi-channel CNN for character-level information capture, and following by a BiLSTM + CRF for sequence labelling and modelling dependencies between the words in the text. We evaluate our model on the benchmark i2b2/2010 dataset, achieving an F1-score of 90.11.
  arXiv  Detail & Related papers  (2023-12-24T21:45:36Z)
• Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs [54.223394825528665]
  We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models. We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
  arXiv  Detail & Related papers  (2023-12-21T14:26:57Z)
• Improving Biomedical Abstractive Summarisation with Knowledge Aggregation from Citation Papers [24.481854035628434]
  Existing language models struggle to generate technical summaries that are on par with those produced by biomedical experts. We propose a novel attention-based citation aggregation model that integrates domain-specific knowledge from citation papers. Our model outperforms state-of-the-art approaches and achieves substantial improvements in abstractive biomedical text summarisation.
  arXiv  Detail & Related papers  (2023-10-24T09:56:46Z)
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)
• BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs [48.376109878173956]
  We present PMC-15M, a novel dataset that is two orders of magnitude larger than existing biomedical multimodal datasets. PMC-15M contains 15 million biomedical image-text pairs collected from 4.4 million scientific articles. Based on PMC-15M, we have pretrained BiomedCLIP, a multimodal foundation model, with domain-specific adaptations tailored to biomedical vision-language processing.
  arXiv  Detail & Related papers  (2023-03-02T02:20:04Z)
• BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining [140.61707108174247]
  We propose BioGPT, a domain-specific generative Transformer language model pre-trained on large scale biomedical literature. We get 44.98%, 38.42% and 40.76% F1 score on BC5CDR, KD-DTI and DDI end-to-end relation extraction tasks respectively, and 78.2% accuracy on PubMedQA.
  arXiv  Detail & Related papers  (2022-10-19T07:17:39Z)
• On the Effectiveness of Compact Biomedical Transformers [12.432191400869002]
  Language models pre-trained on biomedical corpora have recently shown promising results on downstream biomedical tasks. Many existing pre-trained models are resource-intensive and computationally heavy owing to factors such as embedding size, hidden dimension, and number of layers. We introduce six lightweight models, namely, BioDistilBERT, BioTinyBERT, BioMobileBERT, DistilBioBERT, TinyBioBERT, and CompactBioBERT. We evaluate all of our models on three biomedical tasks and compare them with BioBERT-v1.1 to create efficient lightweight models that perform on par with their larger counterparts.
  arXiv  Detail & Related papers  (2022-09-07T14:24:04Z)
• Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processing [55.52858954615655]
  We conduct a systematic study on fine-tuning stability in biomedical NLP. We show that finetuning performance may be sensitive to pretraining settings, especially in low-resource domains. We show that these techniques can substantially improve fine-tuning performance for lowresource biomedical NLP applications.
  arXiv  Detail & Related papers  (2021-12-15T04:20:35Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.