Self-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer

• URL: http://arxiv.org/abs/2110.02950v1
• Date: Wed, 6 Oct 2021 17:57:22 GMT
• Title: Self-Supervised Knowledge Assimilation for Expert-Layman Text Style Transfer
• Authors: Wenda Xu, Michael Saxon, Misha Sra, William Yang Wang
• Abstract summary: Expert-layman text style transfer technologies have the potential to improve communication between scientific communities and the general public. High-quality information produced by experts is often filled with difficult jargon laypeople struggle to understand. This is a particularly notable issue in the medical domain, where layman are often confused by medical text online.
• Score: 63.72621204057025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Expert-layman text style transfer technologies have the potential to improve communication between members of scientific communities and the general public. High-quality information produced by experts is often filled with difficult jargon laypeople struggle to understand. This is a particularly notable issue in the medical domain, where layman are often confused by medical text online. At present, two bottlenecks interfere with the goal of building high-quality medical expert-layman style transfer systems: a dearth of pretrained medical-domain language models spanning both expert and layman terminologies and a lack of parallel corpora for training the transfer task itself. To mitigate the first issue, we propose a novel language model (LM) pretraining task, Knowledge Base Assimilation, to synthesize pretraining data from the edges of a graph of expert- and layman-style medical terminology terms into an LM during self-supervised learning. To mitigate the second issue, we build a large-scale parallel corpus in the medical expert-layman domain using a margin-based criterion. Our experiments show that transformer-based models pretrained on knowledge base assimilation and other well-established pretraining tasks fine-tuning on our new parallel corpus leads to considerable improvement against expert-layman transfer benchmarks, gaining an average relative improvement of our human evaluation, the Overall Success Rate (OSR), by 106%.

Related papers

• ChiMed-GPT: A Chinese Medical Large Language Model with Full Training Regime and Better Alignment to Human Preferences [51.66185471742271]
  We propose ChiMed-GPT, a benchmark LLM designed explicitly for Chinese medical domain. ChiMed-GPT undergoes a comprehensive training regime with pre-training, SFT, and RLHF. We analyze possible biases through prompting ChiMed-GPT to perform attitude scales regarding discrimination of patients.
  arXiv  Detail & Related papers  (2023-11-10T12:25:32Z)
• Large Language Models, scientific knowledge and factuality: A framework to streamline human expert evaluation [0.0]
  This work explores the potential of Large Language Models for dialoguing with biomedical background knowledge. The framework involves of three evaluation steps, each assessing different aspects sequentially: fluency, prompt alignment, semantic coherence, factual knowledge, and specificity of the generated responses. The work provides a systematic assessment on the ability of eleven state-of-the-art models LLMs, including ChatGPT, GPT-4 and Llama 2, in two prompting-based tasks.
  arXiv  Detail & Related papers  (2023-05-28T22:46:21Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• Summarizing Patients Problems from Hospital Progress Notes Using Pre-trained Sequence-to-Sequence Models [9.879960506853145]
  Problem list summarization requires a model to understand, abstract, and generate clinical documentation. We propose a new NLP task that aims to generate a list of problems in a patient's daily care plan using input from the provider's progress notes during hospitalization.
  arXiv  Detail & Related papers  (2022-08-17T17:07:35Z)
• 2021 BEETL Competition: Advancing Transfer Learning for Subject Independence & Heterogenous EEG Data Sets [89.84774119537087]
  We design two transfer learning challenges around diagnostics and Brain-Computer-Interfacing (BCI) Task 1 is centred on medical diagnostics, addressing automatic sleep stage annotation across subjects. Task 2 is centred on Brain-Computer Interfacing (BCI), addressing motor imagery decoding across both subjects and data sets.
  arXiv  Detail & Related papers  (2022-02-14T12:12:20Z)
• Towards more patient friendly clinical notes through language models and ontologies [57.51898902864543]
  We present a novel approach to automated medical text based on word simplification and language modelling. We use a new dataset pairs of publicly available medical sentences and a version of them simplified by clinicians. Our method based on a language model trained on medical forum data generates simpler sentences while preserving both grammar and the original meaning.
  arXiv  Detail & Related papers  (2021-12-23T16:11:19Z)
• Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models [16.535312449449165]
  We release a well-curated biomedical knowledge probing benchmark, MedLAMA, based on the Unified Medical Language System (UMLS) Metathesaurus. We test a wide spectrum of state-of-the-art PLMs and probing approaches on our benchmark, reaching at most 3% of acc@10. We propose Contrastive-Probe, a novel self-supervised contrastive probing approach, that adjusts the underlying PLMs without using any probing data.
  arXiv  Detail & Related papers  (2021-10-15T16:00:11Z)
• Improving Lexically Constrained Neural Machine Translation with Source-Conditioned Masked Span Prediction [6.46964825569749]
  In this paper, we tackle a more challenging setup consisting of domain-specific corpora with much longer n-gram and highly specialized terms. To encourage span-level representations in generation, we additionally impose a source-sentence conditioned masked span prediction loss in the decoder. Experimental results on three domain-specific corpora in two language pairs demonstrate that the proposed training scheme can improve the performance of existing lexically constrained methods.
  arXiv  Detail & Related papers  (2021-05-12T08:11:33Z)
• Automated Lay Language Summarization of Biomedical Scientific Reviews [16.01452242066412]
  Health literacy has emerged as a crucial factor in making appropriate health decisions and ensuring treatment outcomes. Medical jargon and the complex structure of professional language in this domain make health information especially hard to interpret. This paper introduces the novel task of automated generation of lay language summaries of biomedical scientific reviews.
  arXiv  Detail & Related papers  (2020-12-23T10:01:18Z)
• Expertise Style Transfer: A New Task Towards Better Communication between Experts and Laymen [88.30492014778943]
  We propose a new task of expertise style transfer and contribute a manually annotated dataset. Solving this task not only simplifies the professional language, but also improves the accuracy and expertise level of laymen descriptions. We establish the benchmark performance of five state-of-the-art models for style transfer and text simplification.
  arXiv  Detail & Related papers  (2020-05-02T04:50:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.