Conditioned Natural Language Generation using only Unconditioned Language Model: An Exploration

• URL: http://arxiv.org/abs/2011.07347v1
• Date: Sat, 14 Nov 2020 17:45:11 GMT
• Title: Conditioned Natural Language Generation using only Unconditioned Language Model: An Exploration
• Authors: Fan-Keng Sun, Cheng-I Lai
• Abstract summary: Transformer-based language models have shown to be very powerful for natural language generation (NLG) We argue that the original unconditioned LM is sufficient for conditioned NLG. We evaluated our approaches by the samples' fluency and diversity with automated and human evaluation.
• Score: 8.623022983093444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based language models have shown to be very powerful for natural language generation (NLG). However, text generation conditioned on some user inputs, such as topics or attributes, is non-trivial. Past approach relies on either modifying the original LM architecture, re-training the LM on corpora with attribute labels, or having separately trained `guidance models' to guide text generation in decoding. We argued that the above approaches are not necessary, and the original unconditioned LM is sufficient for conditioned NLG. We evaluated our approaches by the samples' fluency and diversity with automated and human evaluation.

Related papers

• Reranking for Natural Language Generation from Logical Forms: A Study based on Large Language Models [47.08364281023261]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language generation. However, their output quality can be inconsistent, posing challenges for generating natural language from logical forms (LFs)
  arXiv  Detail & Related papers  (2023-09-21T17:54:58Z)
• Extrapolating Multilingual Understanding Models as Multilingual Generators [82.1355802012414]
  This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
  arXiv  Detail & Related papers  (2023-05-22T15:33:21Z)
• Word Embeddings Are Steers for Language Models [57.83026781380927]
  We name such steers LM-Steers and find them existing in LMs of all sizes. On tasks such as language model detoxification and sentiment control, LM-Steers can achieve comparable or superior performance. An LM-Steer is transferrable between different language models by an explicit form calculation.
  arXiv  Detail & Related papers  (2023-05-22T07:52:04Z)
• Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models [58.990214815032495]
  Large-scale pre-trained language models have achieved great success on natural language generation tasks. BCLMs have been shown to be efficient in controllable language generation. We propose a "Gemini Discriminator" for controllable language generation which alleviates the mismatch problem with a small computational cost.
  arXiv  Detail & Related papers  (2022-06-11T12:52:32Z)
• Measuring Attribution in Natural Language Generation Models [14.931889185122213]
  We present a new evaluation framework entitled Attributable to Identified Sources (AIS) for assessing the output of natural language generation models. We first define AIS and introduce a two-stage annotation pipeline for allowing annotators to appropriately evaluate model output.
  arXiv  Detail & Related papers  (2021-12-23T22:33:20Z)
• Towards Language Modelling in the Speech Domain Using Sub-word Linguistic Units [56.52704348773307]
  We propose a novel LSTM-based generative speech LM based on linguistic units including syllables and phonemes. With a limited dataset, orders of magnitude smaller than that required by contemporary generative models, our model closely approximates babbling speech. We show the effect of training with auxiliary text LMs, multitask learning objectives, and auxiliary articulatory features.
  arXiv  Detail & Related papers  (2021-10-31T22:48:30Z)
• Multimodal Conditionality for Natural Language Generation [0.0]
  MAnTiS is a general approach for multimodal conditionality in transformer-based Natural Language Generation models. We apply MAnTiS to the task of product description generation, conditioning a network on both product images and titles to generate descriptive text.
  arXiv  Detail & Related papers  (2021-09-02T22:06:07Z)
• Natural Language Generation Using Link Grammar for General Conversational Intelligence [0.0]
  We propose a new technique to automatically generate grammatically valid sentences using the Link Grammar database. This natural language generation method far outperforms current state-of-the-art baselines and may serve as the final component in a proto-AGI question answering pipeline.
  arXiv  Detail & Related papers  (2021-04-19T06:16:07Z)
• Unnatural Language Inference [48.45003475966808]
  We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words. Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
  arXiv  Detail & Related papers  (2020-12-30T20:40:48Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• ETC-NLG: End-to-end Topic-Conditioned Natural Language Generation [1.0152838128195467]
  Plug-and-play language models (PPLMs) enable topic-conditioned natural language generation by pairing large pre-trained generators with attribute models used to steer the predicted token distribution towards the selected topic. We present ETC-NLG, an approach leveraging topic modeling annotations to enable fully-unsupervised End-to-end Topic-Conditioned Natural Language Generation over emergent topics in unlabeled document collections.
  arXiv  Detail & Related papers  (2020-08-25T08:22:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.