ETC-NLG: End-to-end Topic-Conditioned Natural Language Generation

• URL: http://arxiv.org/abs/2008.10875v3
• Date: Tue, 22 Jun 2021 08:45:23 GMT
• Title: ETC-NLG: End-to-end Topic-Conditioned Natural Language Generation
• Authors: Ginevra Carbone, Gabriele Sarti
• Abstract summary: Plug-and-play language models (PPLMs) enable topic-conditioned natural language generation by pairing large pre-trained generators with attribute models used to steer the predicted token distribution towards the selected topic. We present ETC-NLG, an approach leveraging topic modeling annotations to enable fully-unsupervised End-to-end Topic-Conditioned Natural Language Generation over emergent topics in unlabeled document collections.
• Score: 1.0152838128195467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Plug-and-play language models (PPLMs) enable topic-conditioned natural language generation by pairing large pre-trained generators with attribute models used to steer the predicted token distribution towards the selected topic. Despite their computational efficiency, PPLMs require large amounts of labeled texts to effectively balance generation fluency and proper conditioning, making them unsuitable for low-resource settings. We present ETC-NLG, an approach leveraging topic modeling annotations to enable fully-unsupervised End-to-end Topic-Conditioned Natural Language Generation over emergent topics in unlabeled document collections. We first test the effectiveness of our approach in a low-resource setting for Italian, evaluating the conditioning for both topic models and gold annotations. We then perform a comparative evaluation of ETC-NLG for Italian and English using a parallel corpus. Finally, we propose an automatic approach to estimate the effectiveness of conditioning on the generated utterances.

Related papers

• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback [50.84142264245052]
  This work introduces the Align-SLM framework to enhance the semantic understanding of textless Spoken Language Models (SLMs) Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO) We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation.
  arXiv  Detail & Related papers  (2024-11-04T06:07:53Z)
• Preference Alignment Improves Language Model-Based TTS [76.70693823683091]
  preference alignment algorithms adjust LMs to align with the preferences of reward models, enhancing the desirability of the generated content. With a 1.15B parameter LM-based TTS model, we demonstrate that preference alignment consistently improves intelligibility, speaker similarity, and proxy subjective evaluation scores.
  arXiv  Detail & Related papers  (2024-09-19T01:58:19Z)
• A Systematic Evaluation of Large Language Models for Natural Language Generation Tasks [30.54635848057259]
  This paper conducts a comprehensive evaluation of well-known and high-performing large language models (LLMs) We select English and Chinese datasets encompassing Dialogue Generation and Text Summarization. Our study reports both automatic results, accompanied by a detailed analysis.
  arXiv  Detail & Related papers  (2024-05-16T16:56:54Z)
• Plug-and-Play Recipe Generation with Content Planning [28.65323853250831]
  We propose a framework which explicitly models the global content plan of the generated text. It optimize the joint distribution of the natural language sequence and the global content plan in a plug-and-play manner. Our model achieves the state-of-the-art performance on the task of recipe generation.
  arXiv  Detail & Related papers  (2022-12-09T19:39:10Z)
• Bridging the Gap Between Training and Inference of Bayesian Controllable Language Models [58.990214815032495]
  Large-scale pre-trained language models have achieved great success on natural language generation tasks. BCLMs have been shown to be efficient in controllable language generation. We propose a "Gemini Discriminator" for controllable language generation which alleviates the mismatch problem with a small computational cost.
  arXiv  Detail & Related papers  (2022-06-11T12:52:32Z)
• Topic Discovery via Latent Space Clustering of Pretrained Language Model Representations [35.74225306947918]
  We propose a joint latent space learning and clustering framework built upon PLM embeddings. Our model effectively leverages the strong representation power and superb linguistic features brought by PLMs for topic discovery.
  arXiv  Detail & Related papers  (2022-02-09T17:26:08Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Multimodal Conditionality for Natural Language Generation [0.0]
  MAnTiS is a general approach for multimodal conditionality in transformer-based Natural Language Generation models. We apply MAnTiS to the task of product description generation, conditioning a network on both product images and titles to generate descriptive text.
  arXiv  Detail & Related papers  (2021-09-02T22:06:07Z)
• Conditioned Natural Language Generation using only Unconditioned Language Model: An Exploration [8.623022983093444]
  Transformer-based language models have shown to be very powerful for natural language generation (NLG) We argue that the original unconditioned LM is sufficient for conditioned NLG. We evaluated our approaches by the samples' fluency and diversity with automated and human evaluation.
  arXiv  Detail & Related papers  (2020-11-14T17:45:11Z)
• SLM: Learning a Discourse Language Representation with Sentence Unshuffling [53.42814722621715]
  We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation. We show that this feature of our model improves the performance of the original BERT by large margins.
  arXiv  Detail & Related papers  (2020-10-30T13:33:41Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.