A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

• URL: http://arxiv.org/abs/2201.05337v5
• Date: Thu, 24 Aug 2023 08:16:57 GMT
• Title: A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models
• Authors: Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, Dawei Song
• Abstract summary: Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG) We present a systematic critical review on the common tasks, main approaches, and evaluation methods in this area. We discuss the challenges that the field is facing, and put forward various promising future directions.
• Score: 21.124096884958337
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the limited level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks that require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches, and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize the state-of-the-art CTG techniques from the perspective of Transformer-based PLMs. We hope it can help researchers and practitioners in the related fields to quickly track the academic and technological frontier, providing them with a landscape of the area and a roadmap for future research.

Related papers

• Controllable Text Generation for Large Language Models: A Survey [27.110528099257156]
  This paper systematically reviews the latest advancements in Controllable Text Generation for Large Language Models. We categorize CTG tasks into two primary types: content control and control. We address key challenges in current research, including reduced fluency and practicality.
  arXiv  Detail & Related papers  (2024-08-22T17:59:04Z)
• A Comprehensive Survey of Accelerated Generation Techniques in Large Language Models [2.091322528026356]
  This paper presents a survey of accelerated generation techniques in autoregressive language models. We categorize these techniques into several key areas: speculative decoding, early exiting mechanisms, and non-autoregressive methods.
  arXiv  Detail & Related papers  (2024-05-15T07:36:56Z)
• Bringing order into the realm of Transformer-based language models for artificial intelligence and law [3.2074558838636262]
  Transformer-based language models (TLMs) have widely been recognized to be a cutting-edge technology. This article provides the first systematic overview of TLM-based methods for AI-driven problems and tasks in the legal sphere.
  arXiv  Detail & Related papers  (2023-08-10T11:14:22Z)
• Deliberate then Generate: Enhanced Prompting Framework for Text Generation [70.10319005141888]
  Deliberate then Generate (DTG) prompting framework consists of error detection instructions and candidates that may contain errors. We conduct extensive experiments on 20+ datasets across 7 text generation tasks, including summarization, translation, dialogue, and more. We show that DTG consistently outperforms existing prompting methods and achieves state-of-the-art performance on multiple text generation tasks.
  arXiv  Detail & Related papers  (2023-05-31T13:23:04Z)
• Generative Pre-trained Transformer: A Comprehensive Review on Enabling Technologies, Potential Applications, Emerging Challenges, and Future Directions [11.959434388955787]
  The Generative Pre-trained Transformer (GPT) represents a notable breakthrough in the domain of natural language processing. GPT is based on the transformer architecture, a deep neural network designed for natural language processing tasks.
  arXiv  Detail & Related papers  (2023-05-11T19:20:38Z)
• A Survey of Pretrained Language Models Based Text Generation [97.64625999380425]
  Text Generation aims to produce plausible and readable text in human language from input data. Deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs) Grounding text generation on PLMs is seen as a promising direction in both academia and industry.
  arXiv  Detail & Related papers  (2022-01-14T01:44:58Z)
• Pretrained Language Models for Text Generation: A Survey [46.03096493973206]
  We present an overview of the major advances achieved in the topic of pretrained language models (PLMs) for text generation. We discuss how to adapt existing PLMs to model different input data and satisfy special properties in the generated text.
  arXiv  Detail & Related papers  (2021-05-21T12:27:44Z)
• Positioning yourself in the maze of Neural Text Generation: A Task-Agnostic Survey [54.34370423151014]
  This paper surveys the components of modeling approaches relaying task impacts across various generation tasks such as storytelling, summarization, translation etc. We present an abstraction of the imperative techniques with respect to learning paradigms, pretraining, modeling approaches, decoding and the key challenges outstanding in the field in each of them.
  arXiv  Detail & Related papers  (2020-10-14T17:54:42Z)
• Pretrained Transformers for Text Ranking: BERT and Beyond [53.83210899683987]
  This survey provides an overview of text ranking with neural network architectures known as transformers. The combination of transformers and self-supervised pretraining has been responsible for a paradigm shift in natural language processing.
  arXiv  Detail & Related papers  (2020-10-13T15:20:32Z)
• Progressive Generation of Long Text with Pretrained Language Models [83.62523163717448]
  Large-scale language models (LMs) pretrained on massive corpora of text, such as GPT-2, are powerful open-domain text generators. It is still challenging for such models to generate coherent long passages of text, especially when the models are fine-tuned to the target domain on a small corpus. We propose a simple but effective method of generating text in a progressive manner, inspired by generating images from low to high resolution.
  arXiv  Detail & Related papers  (2020-06-28T21:23:05Z)
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer [64.22926988297685]
  Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP) In this paper, we explore the landscape of introducing transfer learning techniques for NLP by a unified framework that converts all text-based language problems into a text-to-text format.
  arXiv  Detail & Related papers  (2019-10-23T17:37:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.