Multi-objective Representation for Numbers in Clinical Narratives Using CamemBERT-bio

• URL: http://arxiv.org/abs/2405.18448v2
• Date: Wed, 10 Jul 2024 08:47:52 GMT
• Title: Multi-objective Representation for Numbers in Clinical Narratives Using CamemBERT-bio
• Authors: Boammani Aser Lompo, Thanh-Dung Le,
• Abstract summary: This research aims to classify numerical values extracted from medical documents across seven physiological categories. We introduce two main innovations: integrating keyword embeddings into the model and adopting a number-agnostic strategy. We show substantial improvements in the effectiveness of CamemBERT-bio, surpassing conventional methods with an F1 score of 0.89.
• Score: 0.9208007322096533
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This research aims to classify numerical values extracted from medical documents across seven distinct physiological categories, employing CamemBERT-bio. Previous studies suggested that transformer-based models might not perform as well as traditional NLP models in such tasks. To enhance CamemBERT-bio's performances, we introduce two main innovations: integrating keyword embeddings into the model and adopting a number-agnostic strategy by excluding all numerical data from the text. The implementation of label embedding techniques refines the attention mechanisms, while the technique of using a `numerical-blind' dataset aims to bolster context-centric learning. Another key component of our research is determining the criticality of extracted numerical data. To achieve this, we utilized a simple approach that involves verifying if the value falls within the established standard ranges. Our findings are encouraging, showing substantial improvements in the effectiveness of CamemBERT-bio, surpassing conventional methods with an F1 score of 0.89. This represents an over 20\% increase over the 0.73 $F_1$ score of traditional approaches and an over 9\% increase over the 0.82 $F_1$ score of state-of-the-art approaches. All this was achieved despite using small and imbalanced training datasets.

Related papers

• Medical-GAT: Cancer Document Classification Leveraging Graph-Based Residual Network for Scenarios with Limited Data [2.913761513290171]
  We present a curated dataset of 1,874 biomedical abstracts, categorized into thyroid cancer, colon cancer, lung cancer, and generic topics. Our research focuses on leveraging this dataset to improve classification performance, particularly in data-scarce scenarios. We introduce a Residual Graph Attention Network (R-GAT) with multiple graph attention layers that capture the semantic information and structural relationships within cancer-related documents.
  arXiv  Detail & Related papers  (2024-10-19T20:07:40Z)
• Depression detection in social media posts using transformer-based models and auxiliary features [6.390468088226495]
  Detection of depression in social media posts is crucial due to the increasing prevalence of mental health issues. Traditional machine learning algorithms often fail to capture intricate textual patterns, limiting their effectiveness in identifying depression. This research proposes a neural network architecture leveraging transformer-based models combined with metadata and linguistic markers.
  arXiv  Detail & Related papers  (2024-09-30T07:53:39Z)
• Improving Extraction of Clinical Event Contextual Properties from Electronic Health Records: A Comparative Study [2.0884301753594334]
  This study performs a comparative analysis of various natural language models for medical text classification. BERT outperforms Bi-LSTM models by up to 28% and the baseline BERT model by up to 16% for recall of the minority classes.
  arXiv  Detail & Related papers  (2024-08-30T10:28:49Z)
• BAL: Balancing Diversity and Novelty for Active Learning [53.289700543331925]
  We introduce a novel framework, Balancing Active Learning (BAL), which constructs adaptive sub-pools to balance diverse and uncertain data. Our approach outperforms all established active learning methods on widely recognized benchmarks by 1.20%.
  arXiv  Detail & Related papers  (2023-12-26T08:14:46Z)
• Multi-level biomedical NER through multi-granularity embeddings and enhanced labeling [3.8599767910528917]
  This paper proposes a hybrid approach that integrates the strengths of multiple models. BERT provides contextualized word embeddings, a pre-trained multi-channel CNN for character-level information capture, and following by a BiLSTM + CRF for sequence labelling and modelling dependencies between the words in the text. We evaluate our model on the benchmark i2b2/2010 dataset, achieving an F1-score of 90.11.
  arXiv  Detail & Related papers  (2023-12-24T21:45:36Z)
• Diversifying Knowledge Enhancement of Biomedical Language Models using Adapter Modules and Knowledge Graphs [54.223394825528665]
  We develop an approach that uses lightweight adapter modules to inject structured biomedical knowledge into pre-trained language models. We use two large KGs, the biomedical knowledge system UMLS and the novel biochemical OntoChem, with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. We show that our methodology leads to performance improvements in several instances while keeping requirements in computing power low.
  arXiv  Detail & Related papers  (2023-12-21T14:26:57Z)
• Improving Biomedical Entity Linking with Retrieval-enhanced Learning [53.24726622142558]
  $k$NN-BioEL provides a BioEL model with the ability to reference similar instances from the entire training corpus as clues for prediction. We show that $k$NN-BioEL outperforms state-of-the-art baselines on several datasets.
  arXiv  Detail & Related papers  (2023-12-15T14:04:23Z)
• Using Large Language Models to Automate Category and Trend Analysis of Scientific Articles: An Application in Ophthalmology [4.455826633717872]
  We present an automated method for article classification, leveraging the power of Large Language Models (LLM) The model achieved mean accuracy of 0.86 and mean F1 of 0.85 based on the RenD dataset. The extendibility of the model to other scientific fields broadens its impact in facilitating research and trend analysis across diverse disciplines.
  arXiv  Detail & Related papers  (2023-08-31T12:45:53Z)
• CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data [1.1265248232450553]
  Transfer learning with BERT-like models has allowed major advances for French, especially for named entity recognition. We introduce CamemBERT-bio, a dedicated French biomedical model derived from a new public French biomedical dataset. Through continual pre-training, CamemBERT-bio achieves an improvement of 2.54 points of F1-score on average across various biomedical named entity recognition tasks.
  arXiv  Detail & Related papers  (2023-06-27T15:23:14Z)
• From Zero to Hero: Harnessing Transformers for Biomedical Named Entity Recognition in Zero- and Few-shot Contexts [0.0]
  This paper proposes a method for zero- and few-shot NER in the biomedical domain. We have achieved average F1 scores of 35.44% for zero-shot NER, 50.10% for one-shot NER, 69.94% for 10-shot NER, and 79.51% for 100-shot NER on 9 diverse evaluated biomedical entities.
  arXiv  Detail & Related papers  (2023-05-05T12:14:22Z)
• TWINS: A Fine-Tuning Framework for Improved Transferability of Adversarial Robustness and Generalization [89.54947228958494]
  This paper focuses on the fine-tuning of an adversarially pre-trained model in various classification tasks. We propose a novel statistics-based approach, Two-WIng NormliSation (TWINS) fine-tuning framework. TWINS is shown to be effective on a wide range of image classification datasets in terms of both generalization and robustness.
  arXiv  Detail & Related papers  (2023-03-20T14:12:55Z)
• Clinical Concept and Relation Extraction Using Prompt-based Machine Reading Comprehension [38.79665143111312]
  We formulate both clinical concept extraction and relation extraction using a unified prompt-based machine reading comprehension architecture. We compare our MRC models with existing deep learning models for concept extraction and end-to-end relation extraction. We evaluate the transfer learning ability of the proposed MRC models in a cross-institution setting.
  arXiv  Detail & Related papers  (2023-03-14T22:37:31Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• Drug Synergistic Combinations Predictions via Large-Scale Pre-Training and Graph Structure Learning [82.93806087715507]
  Drug combination therapy is a well-established strategy for disease treatment with better effectiveness and less safety degradation. Deep learning models have emerged as an efficient way to discover synergistic combinations. Our framework achieves state-of-the-art results in comparison with other deep learning-based methods.
  arXiv  Detail & Related papers  (2023-01-14T15:07:43Z)
• A Meta-GNN approach to personalized seizure detection and classification [53.906130332172324]
  We propose a personalized seizure detection and classification framework that quickly adapts to a specific patient from limited seizure samples. We train a Meta-GNN based classifier that learns a global model from a set of training patients. We show that our method outperforms the baselines by reaching 82.7% on accuracy and 82.08% on F1 score after only 20 iterations on new unseen patients.
  arXiv  Detail & Related papers  (2022-11-01T14:12:58Z)
• Deeper Clinical Document Understanding Using Relation Extraction [0.0]
  We propose a text mining framework comprising of Named Entity Recognition (NER) and Relation Extraction (RE) models. We introduce two new RE model architectures -- an accuracy-optimized one based on BioBERT and a speed-optimized one utilizing crafted features over a Fully Connected Neural Network (FCNN) We show two practical applications of this framework -- for building a biomedical knowledge graph and for improving the accuracy of mapping entities to clinical codes.
  arXiv  Detail & Related papers  (2021-12-25T17:14:13Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• Select-ProtoNet: Learning to Select for Few-Shot Disease Subtype Prediction [55.94378672172967]
  We focus on few-shot disease subtype prediction problem, identifying subgroups of similar patients. We introduce meta learning techniques to develop a new model, which can extract the common experience or knowledge from interrelated clinical tasks. Our new model is built upon a carefully designed meta-learner, called Prototypical Network, that is a simple yet effective meta learning machine for few-shot image classification.
  arXiv  Detail & Related papers  (2020-09-02T02:50:30Z)
• Predicting Clinical Diagnosis from Patients Electronic Health Records Using BERT-based Neural Networks [62.9447303059342]
  We show the importance of this problem in medical community. We present a modification of Bidirectional Representations from Transformers (BERT) model for classification sequence. We use a large-scale Russian EHR dataset consisting of about 4 million unique patient visits.
  arXiv  Detail & Related papers  (2020-07-15T09:22:55Z)
• Students Need More Attention: BERT-based AttentionModel for Small Data with Application to AutomaticPatient Message Triage [65.7062363323781]
  We propose a novel framework based on BioBERT (Bidirectional Representations from Transformers forBiomedical TextMining) We introduce Label Embeddings for Self-Attention in each layer of BERT, which we call LESA-BERT, and (ii) by distilling LESA-BERT to smaller variants, we aim to reduce overfitting and model size when working on small datasets. As an application, our framework is utilized to build a model for patient portal message triage that classifies the urgency of a message into three categories: non-urgent, medium and urgent.
  arXiv  Detail & Related papers  (2020-06-22T03:39:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.