Improving Predictions of Tail-end Labels using Concatenated BioMed-Transformers for Long Medical Documents

• URL: http://arxiv.org/abs/2112.01718v1
• Date: Fri, 3 Dec 2021 05:06:43 GMT
• Title: Improving Predictions of Tail-end Labels using Concatenated BioMed-Transformers for Long Medical Documents
• Authors: Vithya Yogarajan, Bernhard Pfahringer, Tony Smith, Jacob Montiel
• Abstract summary: This research aims to improve F1 scores of infrequent labels across multi-label problems, especially with long-tail labels. New state-of-the-art (SOTA) results are obtained using TransformerXL for predicting medical codes.
• Score: 3.0625089376654664
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-label learning predicts a subset of labels from a given label set for an unseen instance while considering label correlations. A known challenge with multi-label classification is the long-tailed distribution of labels. Many studies focus on improving the overall predictions of the model and thus do not prioritise tail-end labels. Improving the tail-end label predictions in multi-label classifications of medical text enables the potential to understand patients better and improve care. The knowledge gained by one or more infrequent labels can impact the cause of medical decisions and treatment plans. This research presents variations of concatenated domain-specific language models, including multi-BioMed-Transformers, to achieve two primary goals. First, to improve F1 scores of infrequent labels across multi-label problems, especially with long-tail labels; second, to handle long medical text and multi-sourced electronic health records (EHRs), a challenging task for standard transformers designed to work on short input sequences. A vital contribution of this research is new state-of-the-art (SOTA) results obtained using TransformerXL for predicting medical codes. A variety of experiments are performed on the Medical Information Mart for Intensive Care (MIMIC-III) database. Results show that concatenated BioMed-Transformers outperform standard transformers in terms of overall micro and macro F1 scores and individual F1 scores of tail-end labels, while incurring lower training times than existing transformer-based solutions for long input sequences.

Related papers

• FedMLP: Federated Multi-Label Medical Image Classification under Task Heterogeneity [30.49607763632271]
  Cross-silo federated learning (FL) enables decentralized organizations to collaboratively train models while preserving data privacy. We propose a two-stage method FedMLP to combat class missing from two aspects: pseudo label tagging and global knowledge learning. Experiments on two publicly-available medical datasets validate the superiority of FedMLP against the state-of-the-art both federated semi-supervised and noisy label learning approaches.
  arXiv  Detail & Related papers  (2024-06-27T08:36:43Z)
• Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification [6.0233642055651115]
  We introduce Medformer, a multi-granularity patching transformer tailored specifically for medical time series classification. Our method incorporates three novel mechanisms to leverage the unique characteristics of medical time series.
  arXiv  Detail & Related papers  (2024-05-24T16:51:10Z)
• Biomedical Entity Linking as Multiple Choice Question Answering [48.74212158495695]
  We present BioELQA, a novel model that treats Biomedical Entity Linking as Multiple Choice Question Answering. We first obtains candidate entities with a fast retriever, jointly presents the mention and candidate entities to a generator, and then outputs the predicted symbol associated with its chosen entity. To improve generalization for long-tailed entities, we retrieve similar labeled training instances as clues and the input with retrieved instances for the generator.
  arXiv  Detail & Related papers  (2024-02-23T08:40:38Z)
• DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Prediction [4.94752151268185]
  This paper presents DKEC, Domain Knowledge Enhanced Classification for diagnosis prediction. We construct DKEC using three online medical knowledge sources and evaluate it on a real-world Emergency Medical Services (EMS) dataset and a public electronic health record (EHR) dataset. Results show that DKEC outperforms the state-of-the-art label-wise attention networks and transformer models of different sizes.
  arXiv  Detail & Related papers  (2023-10-10T22:53:15Z)
• Multi-label Classification with High-rank and High-order Label Correlations [62.39748565407201]
  Previous methods capture the high-order label correlations mainly by transforming the label matrix to a latent label space with low-rank matrix factorization. We propose a simple yet effective method to depict the high-order label correlations explicitly, and at the same time maintain the high-rank of the label matrix. Comparative studies over twelve benchmark data sets validate the effectiveness of the proposed algorithm in multi-label classification.
  arXiv  Detail & Related papers  (2022-07-09T05:15:31Z)
• Statistical Dependency Guided Contrastive Learning for Multiple Labeling in Prenatal Ultrasound [56.631021151764955]
  Standard plane recognition plays an important role in prenatal ultrasound (US) screening. We build a novel multi-label learning scheme to identify multiple standard planes and corresponding anatomical structures simultaneously.
  arXiv  Detail & Related papers  (2021-08-11T06:39:26Z)
• Modeling Diagnostic Label Correlation for Automatic ICD Coding [37.79764232289666]
  We propose a two-stage framework to improve automatic ICD coding by capturing the label correlation. Specifically, we train a label set distribution estimator to rescore the probability of each label set candidate. In the experiments, our proposed framework is able to improve upon best-performing predictors on the benchmark MIMIC datasets.
  arXiv  Detail & Related papers  (2021-06-24T07:26:30Z)
• A Study on the Autoregressive and non-Autoregressive Multi-label Learning [77.11075863067131]
  We propose a self-attention based variational encoder-model to extract the label-label and label-feature dependencies jointly. Our model can therefore be used to predict all labels in parallel while still including both label-label and label-feature dependencies.
  arXiv  Detail & Related papers  (2020-12-03T05:41:44Z)
• Collaborative Unsupervised Domain Adaptation for Medical Image Diagnosis [102.40869566439514]
  We seek to exploit rich labeled data from relevant domains to help the learning in the target task via Unsupervised Domain Adaptation (UDA) Unlike most UDA methods that rely on clean labeled data or assume samples are equally transferable, we innovatively propose a Collaborative Unsupervised Domain Adaptation algorithm. We theoretically analyze the generalization performance of the proposed method, and also empirically evaluate it on both medical and general images.
  arXiv  Detail & Related papers  (2020-07-05T11:49:17Z)
• Interaction Matching for Long-Tail Multi-Label Classification [57.262792333593644]
  We present an elegant and effective approach for addressing limitations in existing multi-label classification models. By performing soft n-gram interaction matching, we match labels with natural language descriptions.
  arXiv  Detail & Related papers  (2020-05-18T15:27:55Z)
• Seeing The Whole Patient: Using Multi-Label Medical Text Classification Techniques to Enhance Predictions of Medical Codes [2.158285012874102]
  We present results of multi-label medical text classification problems with 18, 50 and 155 labels. For imbalanced data we show that labels which occur infrequently, benefit the most from additional features incorporated in embeddings. High dimensional embeddings from this research are made available for public use.
  arXiv  Detail & Related papers  (2020-03-29T02:19:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.