Federated pretraining and fine tuning of BERT using clinical notes from multiple silos

• URL: http://arxiv.org/abs/2002.08562v1
• Date: Thu, 20 Feb 2020 04:14:35 GMT
• Title: Federated pretraining and fine tuning of BERT using clinical notes from multiple silos
• Authors: Dianbo Liu, Tim Miller
• Abstract summary: We show that it is possible to pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.
• Score: 4.794677806040309
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large scale contextual representation models, such as BERT, have significantly advanced natural language processing (NLP) in recently years. However, in certain area like healthcare, accessing diverse large scale text data from multiple institutions is extremely challenging due to privacy and regulatory reasons. In this article, we show that it is possible to both pretrain and fine tune BERT models in a federated manner using clinical texts from different silos without moving the data.

Related papers

• Romanization-based Large-scale Adaptation of Multilingual Language Models [124.57923286144515]
  Large multilingual pretrained language models (mPLMs) have become the de facto state of the art for cross-lingual transfer in NLP. We study and compare a plethora of data- and parameter-efficient strategies for adapting the mPLMs to romanized and non-romanized corpora of 14 diverse low-resource languages. Our results reveal that UROMAN-based transliteration can offer strong performance for many languages, with particular gains achieved in the most challenging setups.
  arXiv  Detail & Related papers  (2023-04-18T09:58:34Z)
• Lightweight Transformers for Clinical Natural Language Processing [9.532776962985828]
  This study focuses on development of compact language models for processing clinical texts. We developed a number of efficient lightweight clinical transformers using knowledge distillation and continual learning. Our evaluation was done across several standard datasets and covered a wide range of clinical text-mining tasks.
  arXiv  Detail & Related papers  (2023-02-09T16:07:31Z)
• An Interpretable End-to-end Fine-tuning Approach for Long Clinical Text [72.62848911347466]
  Unstructured clinical text in EHRs contains crucial information for applications including decision support, trial matching, and retrospective research. Recent work has applied BERT-based models to clinical information extraction and text classification, given these models' state-of-the-art performance in other NLP domains. In this work, we propose a novel fine-tuning approach called SnipBERT. Instead of using entire notes, SnipBERT identifies crucial snippets and feeds them into a truncated BERT-based model in a hierarchical manner.
  arXiv  Detail & Related papers  (2020-11-12T17:14:32Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing [73.37262264915739]
  We show that for domains with abundant unlabeled text, such as biomedicine, pretraining language models from scratch results in substantial gains. Our experiments show that domain-specific pretraining serves as a solid foundation for a wide range of biomedical NLP tasks.
  arXiv  Detail & Related papers  (2020-07-31T00:04:15Z)
• Pre-training technique to localize medical BERT and enhance biomedical BERT [0.0]
  It is difficult to train specific BERT models that perform well for domains in which there are few publicly available databases of high quality and large size. We propose a single intervention with one option: simultaneous pre-training after up-sampling and amplified vocabulary. Our Japanese medical BERT outperformed conventional baselines and the other BERT models in terms of the medical document classification task.
  arXiv  Detail & Related papers  (2020-05-14T18:00:01Z)
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT [60.9051207862378]
  multilingual BERT works remarkably well on cross-lingual transfer tasks. Datasize and context window size are crucial factors to the transferability. There is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
  arXiv  Detail & Related papers  (2020-04-20T11:13:16Z)
• Sensitive Data Detection and Classification in Spanish Clinical Text: Experiments with BERT [0.8379286663107844]
  In this paper, we use a BERT-based sequence labelling model to conduct anonymisation experiments in Spanish. Experiments show that a simple BERT-based model with general-domain pre-training obtains highly competitive results without any domain specific feature engineering.
  arXiv  Detail & Related papers  (2020-03-06T09:46:51Z)
• What BERT Sees: Cross-Modal Transfer for Visual Question Generation [21.640299110619384]
  We study the visual capabilities of BERT out-of-the-box, by avoiding pre-training made on supplementary data. We introduce BERT-gen, a BERT-based architecture for text generation, able to leverage on either mono- or multi- modal representations.
  arXiv  Detail & Related papers  (2020-02-25T12:44:36Z)
• Multilingual Denoising Pre-training for Neural Machine Translation [132.66750663226287]
  mBART is a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora. mBART is one of the first methods for pre-training a complete sequence-to-sequence model.
  arXiv  Detail & Related papers  (2020-01-22T18:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.