COLLIE: Systematic Construction of Constrained Text Generation Tasks

• URL: http://arxiv.org/abs/2307.08689v1
• Date: Mon, 17 Jul 2023 17:48:51 GMT
• Title: COLLIE: Systematic Construction of Constrained Text Generation Tasks
• Authors: Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, Karthik Narasimhan
• Abstract summary: COLLIE is a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels. We develop tools for automatic extraction of task instances given a constraint structure and a raw text corpus. We perform systematic experiments across five state-of-the-art instruction-tuned language models and analyze their performances to reveal shortcomings.
• Score: 33.300039566331876
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text generation under constraints have seen increasing interests in natural language processing, especially with the rapidly improving capabilities of large language models. However, existing benchmarks for constrained generation usually focus on fixed constraint types (e.g.,generate a sentence containing certain words) that have proved to be easy for state-of-the-art models like GPT-4. We present COLLIE, a grammar-based framework that allows the specification of rich, compositional constraints with diverse generation levels (word, sentence, paragraph, passage) and modeling challenges (e.g.,language understanding, logical reasoning, counting, semantic planning). We also develop tools for automatic extraction of task instances given a constraint structure and a raw text corpus. Using COLLIE, we compile the COLLIE-v1 dataset with 2080 instances comprising 13 constraint structures. We perform systematic experiments across five state-of-the-art instruction-tuned language models and analyze their performances to reveal shortcomings. COLLIE is designed to be extensible and lightweight, and we hope the community finds it useful to develop more complex constraints and evaluations in the future.

Related papers

• Controllable Text Generation in the Instruction-Tuning Era [3.310278632293704]
  We find that prompting-based approaches outperform controllable text generation methods on most datasets and tasks. We provide an algorithm that uses only a task dataset and a Large Language Model with in-context capabilities to automatically generate a constraint dataset.
  arXiv  Detail & Related papers  (2024-05-02T17:24:30Z)
• Language Models for Text Classification: Is In-Context Learning Enough? [54.869097980761595]
  Recent foundational language models have shown state-of-the-art performance in many NLP tasks in zero- and few-shot settings. An advantage of these models over more standard approaches is the ability to understand instructions written in natural language (prompts) This makes them suitable for addressing text classification problems for domains with limited amounts of annotated instances.
  arXiv  Detail & Related papers  (2024-03-26T12:47:39Z)
• Instruct-SCTG: Guiding Sequential Controlled Text Generation through Instructions [42.67608830386934]
  Instruct-SCTG is a sequential framework that harnesses instruction-tuned language models to generate structurally coherent text. Our framework generates articles in a section-by-section manner, aligned with the desired human structure using natural language instructions.
  arXiv  Detail & Related papers  (2023-12-19T16:20:49Z)
• Toward Unified Controllable Text Generation via Regular Expression Instruction [56.68753672187368]
  Our paper introduces Regular Expression Instruction (REI), which utilizes an instruction-based mechanism to fully exploit regular expressions' advantages to uniformly model diverse constraints. Our method only requires fine-tuning on medium-scale language models or few-shot, in-context learning on large language models, and requires no further adjustment when applied to various constraint combinations.
  arXiv  Detail & Related papers  (2023-09-19T09:05:14Z)
• Efficient Guided Generation for Large Language Models [0.21485350418225244]
  We show how the problem of neural text generation can be constructively reformulated in terms of transitions between the states of a finite-state machine. This framework leads to an efficient approach to guiding text generation with regular expressions and context-free grammars.
  arXiv  Detail & Related papers  (2023-07-19T01:14:49Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings is a significant challenge. We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences. We demonstrate that generative models like GPT can accurately learn this CFG language and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• Pre-Training to Learn in Context [138.0745138788142]
  The ability of in-context learning is not fully exploited because language models are not explicitly trained to learn in context. We propose PICL (Pre-training for In-Context Learning), a framework to enhance the language models' in-context learning ability. Our experiments show that PICL is more effective and task-generalizable than a range of baselines, outperforming larger language models with nearly 4x parameters.
  arXiv  Detail & Related papers  (2023-05-16T03:38:06Z)
• Controlled Text Generation with Natural Language Instructions [74.88938055638636]
  InstructCTG is a controlled text generation framework that incorporates different constraints. We first extract the underlying constraints of natural texts through a combination of off-the-shelf NLP tools and simple verbalizes. By prepending natural language descriptions of the constraints and a few demonstrations, we fine-tune a pre-trained language model to incorporate various types of constraints.
  arXiv  Detail & Related papers  (2023-04-27T15:56:34Z)
• Controllable Text Generation with Language Constraints [39.741059642044874]
  We consider the task of text generation in language models with constraints specified in natural language. Our benchmark contains knowledge-intensive constraints sourced from databases like Wordnet and Wikidata. We propose a solution to leverage a language model's own internal knowledge to guide generation.
  arXiv  Detail & Related papers  (2022-12-20T17:39:21Z)
• Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence [59.51720326054546]
  We propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. Our model can generate more coherent texts than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-05-19T07:29:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.