Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification

• URL: http://arxiv.org/abs/2305.12463v1
• Date: Sun, 21 May 2023 14:03:49 GMT
• Title: Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification
• Authors: Renliang Sun, Wei Xu, Xiaojun Wan
• Abstract summary: Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. We propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts.
• Score: 59.625179404482594
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Randomly masking text spans in ordinary texts in the pre-training stage hardly allows models to acquire the ability to generate simple texts. It can hurt the performance of pre-trained models on text simplification tasks. In this paper, we propose a new continued pre-training strategy to teach the pre-trained model to generate simple texts. We continue pre-training BART, a representative model, to obtain SimpleBART. It consistently and significantly improves the results on lexical simplification, sentence simplification, and document-level simplification tasks over BART. At the end, we compare SimpleBART with several representative large language models (LLMs).

Related papers

• Controlling Pre-trained Language Models for Grade-Specific Text Simplification [22.154454849167077]
  We study how different control mechanisms impact the adequacy and simplicity of text simplification systems. We introduce a simple method that predicts the edit operations required for simplifying a text for a specific grade level on an instance-per-instance basis.
  arXiv  Detail & Related papers  (2023-05-24T10:29:45Z)
• Classifiers are Better Experts for Controllable Text Generation [63.17266060165098]
  We show that the proposed method significantly outperforms recent PPLM, GeDi, and DExperts on PPL and sentiment accuracy based on the external classifier of generated texts. The same time, it is also easier to implement and tune, and has significantly fewer restrictions and requirements.
  arXiv  Detail & Related papers  (2022-05-15T12:58:35Z)
• SimpleBERT: A Pre-trained Model That Learns to Generate Simple Words [59.142185753887645]
  In this work, we propose a continued pre-training method for text simplification. We use a small-scale simple text dataset for continued pre-training and employ two methods to identify simple words. We obtain SimpleBERT, which surpasses BERT in both lexical simplification and sentence simplification tasks.
  arXiv  Detail & Related papers  (2022-04-16T11:28:01Z)
• Automatic Lexical Simplification for Turkish [0.0]
  We present the first automatic lexical simplification system for the Turkish language. Recent text simplification efforts rely on manually crafted simplified corpora and comprehensive NLP tools. We present a new text simplification pipeline based on pretrained representation model BERT together with morphological features to generate grammatically correct and semantically appropriate word-level simplifications.
  arXiv  Detail & Related papers  (2022-01-15T15:58:44Z)
• LAFITE: Towards Language-Free Training for Text-to-Image Generation [83.2935513540494]
  We propose the first work to train text-to-image generation models without any text data. Our method leverages the well-aligned multi-modal semantic space of the powerful pre-trained CLIP model. We obtain state-of-the-art results in the standard text-to-image generation tasks.
  arXiv  Detail & Related papers  (2021-11-27T01:54:45Z)
• Few-Shot Text Generation with Pattern-Exploiting Training [12.919486518128734]
  In this paper, we show that the underlying idea can also be applied to text generation tasks. We adapt Pattern-Exploiting Training (PET), a recently proposed few-shot approach, for finetuning generative language models on text generation tasks.
  arXiv  Detail & Related papers  (2020-12-22T10:53:07Z)
• Explainable Prediction of Text Complexity: The Missing Preliminaries for Text Simplification [13.447565774887215]
  Text simplification reduces the language complexity of professional content for accessibility purposes. End-to-end neural network models have been widely adopted to directly generate the simplified version of input text. We show that text simplification can be decomposed into a compact pipeline of tasks to ensure the transparency and explainability of the process.
  arXiv  Detail & Related papers  (2020-07-31T03:33:37Z)
• Progressive Generation of Long Text with Pretrained Language Models [83.62523163717448]
  Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. It is still challenging for such models to generate coherent long passages of text, especially when the models are fine-tuned to the target domain on a small corpus. We propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution.
  arXiv  Detail & Related papers  (2020-06-28T21:23:05Z)
• Multilingual Denoising Pre-training for Neural Machine Translation [132.66750663226287]
  mBART is a sequence-to-sequence denoising auto-encoder pre-trained on large-scale monolingual corpora. mBART is one of the first methods for pre-training a complete sequence-to-sequence model.
  arXiv  Detail & Related papers  (2020-01-22T18:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.