Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio

• URL: http://arxiv.org/abs/2306.15926v1
• Date: Wed, 28 Jun 2023 05:10:51 GMT
• Title: Most Language Models can be Poets too: An AI Writing Assistant and Constrained Text Generation Studio
• Authors: Allen Roush, Sanjay Basu, Akshay Moorthy, Dmitry Dubovoy
• Abstract summary: We find that most language models generate compelling text even under significant constraints. We present a technique for modifying the output of a language model by compositionally applying filter functions to the language models vocabulary. We also present a Huggingface space web-app presenting this technique called Gadsby.
• Score: 0.5097809301149341
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite rapid advancement in the field of Constrained Natural Language Generation, little time has been spent on exploring the potential of language models which have had their vocabularies lexically, semantically, and/or phonetically constrained. We find that most language models generate compelling text even under significant constraints. We present a simple and universally applicable technique for modifying the output of a language model by compositionally applying filter functions to the language models vocabulary before a unit of text is generated. This approach is plug-and-play and requires no modification to the model. To showcase the value of this technique, we present an easy to use AI writing assistant called Constrained Text Generation Studio (CTGS). CTGS allows users to generate or choose from text with any combination of a wide variety of constraints, such as banning a particular letter, forcing the generated words to have a certain number of syllables, and/or forcing the words to be partial anagrams of another word. We introduce a novel dataset of prose that omits the letter e. We show that our method results in strictly superior performance compared to fine-tuning alone on this dataset. We also present a Huggingface space web-app presenting this technique called Gadsby. The code is available to the public here: https://github.com/Hellisotherpeople/Constrained-Text-Generation-Studio

Related papers

• Learning to Generate Text in Arbitrary Writing Styles [6.7308816341849695]
  It is desirable for language models to produce text in an author-specific style on the basis of a potentially small writing sample. We propose to guide a language model to generate text in a target style using contrastively-trained representations that capture stylometric features.
  arXiv  Detail & Related papers  (2023-12-28T18:58:52Z)
• TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering [118.30923824681642]
  TextDiffuser-2 aims to unleash the power of language models for text rendering. We utilize the language model within the diffusion model to encode the position and texts at the line level. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V.
  arXiv  Detail & Related papers  (2023-11-28T04:02:40Z)
• Copy Is All You Need [66.00852205068327]
  We formulate text generation as progressively copying text segments from an existing text collection. Our approach achieves better generation quality according to both automatic and human evaluations. Our approach attains additional performance gains by simply scaling up to larger text collections.
  arXiv  Detail & Related papers  (2023-07-13T05:03:26Z)
• Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation [84.95981645040281]
  Planning can be a useful intermediate step to render conditional generation less opaque and more grounded. We present a web browser-based demonstration for query-focused summarization that uses a sequence of question-answer pairs.
  arXiv  Detail & Related papers  (2023-04-28T18:14:48Z)
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP [22.772546707304766]
  We show how hybrid approaches of words and characters as well as subword-based approaches based on learned segmentation have been proposed and evaluated. We conclude that there is and likely will never be a silver bullet singular solution for all applications.
  arXiv  Detail & Related papers  (2021-12-20T13:04:18Z)
• A Plug-and-Play Method for Controlled Text Generation [38.283313068622085]
  We present a plug-and-play decoding method for controlled language generation that is so simple and intuitive, it can be described in a single sentence. Despite the simplicity of this approach, we see it works incredibly well in practice.
  arXiv  Detail & Related papers  (2021-09-20T17:27:03Z)
• Revisiting Language Encoding in Learning Multilingual Representations [70.01772581545103]
  We propose a new approach called Cross-lingual Language Projection (XLP) to replace language embedding. XLP projects the word embeddings into language-specific semantic space, and then the projected embeddings will be fed into the Transformer model. Experiments show that XLP can freely and significantly boost the model performance on extensive multilingual benchmark datasets.
  arXiv  Detail & Related papers  (2021-02-16T18:47:10Z)
• Directed Beam Search: Plug-and-Play Lexically Constrained Language Generation [6.2211479935811775]
  State-of-the-art language models are too large to be trained from scratch in a manageable time. We propose Directed Beam Search (DBS), a plug-and-play method for lexically constrained language generation.
  arXiv  Detail & Related papers  (2020-12-31T03:05:44Z)
• Breaking Writer's Block: Low-cost Fine-tuning of Natural Language Generation Models [62.997667081978825]
  We describe a system that fine-tunes a natural language generation model for the problem of solving Writer's Block. The proposed fine-tuning obtains excellent results, even with a small number of epochs and a total cost of USD 150.
  arXiv  Detail & Related papers  (2020-12-19T11:19:11Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.