DTPA: Dynamic Token-level Prefix Augmentation for Controllable Text Generation

• URL: http://arxiv.org/abs/2508.04047v1
• Date: Wed, 06 Aug 2025 03:20:33 GMT
• Title: DTPA: Dynamic Token-level Prefix Augmentation for Controllable Text Generation
• Authors: Jiabing Yang, Yixiang Chen, Zichen Wen, Chenhang Cui, Peiyan Li, Yuan Xu, Bowen Fang, Yan Huang, Liang Wang,
• Abstract summary: Controllability of texts generated by Air-Decoding tends to decline with increasing sequence length.<n>Different types of prefixes including soft and hard prefixes are also key factors influencing performance.<n>We propose Dynamic Token-level Prefix Augmentation (DTPA) based on Air-Decoding for controllable text generation.
• Score: 10.166289358891238
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controllable Text Generation (CTG) is a vital subfield in Natural Language Processing (NLP), aiming to generate text that aligns with desired attributes. However, previous studies commonly focus on the quality of controllable text generation for short sequences, while the generation of long-form text remains largely underexplored. In this paper, we observe that the controllability of texts generated by the powerful prefix-based method Air-Decoding tends to decline with increasing sequence length, which we hypothesize primarily arises from the observed decay in attention to the prefixes. Meanwhile, different types of prefixes including soft and hard prefixes are also key factors influencing performance. Building on these insights, we propose a lightweight and effective framework called Dynamic Token-level Prefix Augmentation (DTPA) based on Air-Decoding for controllable text generation. Specifically, it first selects the optimal prefix type for a given task. Then we dynamically amplify the attention to the prefix for the attribute distribution to enhance controllability, with a scaling factor growing exponentially as the sequence length increases. Moreover, based on the task, we optionally apply a similar augmentation to the original prompt for the raw distribution to balance text quality. After attribute distribution reconstruction, the generated text satisfies the attribute constraints well. Experiments on multiple CTG tasks demonstrate that DTPA generally outperforms other methods in attribute control while maintaining competitive fluency, diversity, and topic relevance. Further analysis highlights DTPA's superior effectiveness in long text generation.

Related papers

• Attribute First, then Generate: Locally-attributable Grounded Text Generation [33.371400233333326]
  We introduce a locally-attributable text generation approach, prioritizing concise attributions. Our method, named "Attribute First, then Generate", breaks down the conventional end-to-end generation process into three intuitive steps.
  arXiv  Detail & Related papers  (2024-03-25T18:41:47Z)
• Harnessing the Plug-and-Play Controller by Prompting [12.705251690623495]
  This paper introduces a novel method for flexible attribute control in text generation using pre-trained language models (PLMs) The proposed approach aims to enhance the fluency of generated text by guiding the generation process with PPCs.
  arXiv  Detail & Related papers  (2024-02-06T17:18:25Z)
• Successor Features for Efficient Multisubject Controlled Text Generation [48.37713738712319]
  We introduce SF-GEN, which is grounded in two primary concepts: successor features (SFs) and language model rectification. SF-GEN seamlessly integrates the two to enable dynamic steering of text generation with no need to alter the LLM's parameters. To the best of our knowledge, our research represents the first application of successor features in text generation.
  arXiv  Detail & Related papers  (2023-11-03T00:17:08Z)
• Air-Decoding: Attribute Distribution Reconstruction for Decoding-Time Controllable Text Generation [58.911255139171075]
  Controllable text generation (CTG) aims to generate text with desired attributes. We propose a novel lightweight decoding framework named Air-Decoding. Our method achieves a new state-of-the-art control performance.
  arXiv  Detail & Related papers  (2023-10-23T12:59:11Z)
• Optimizing Factual Accuracy in Text Generation through Dynamic Knowledge Selection [71.20871905457174]
  Language models (LMs) have revolutionized the way we interact with information, but they often generate nonfactual text. Previous methods use external knowledge as references for text generation to enhance factuality but often struggle with the knowledge mix-up of irrelevant references. We present DKGen, which divide the text generation process into an iterative process.
  arXiv  Detail & Related papers  (2023-08-30T02:22:40Z)
• TPS++: Attention-Enhanced Thin-Plate Spline for Scene Text Recognition [78.67283660198403]
  Text irregularities pose significant challenges to scene text recognizers. TPS++ is an attention-enhanced TPS transformation that incorporates the attention mechanism to text rectification. It consistently improves the recognition and achieves state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2023-05-09T10:16:43Z)
• Sequentially Controlled Text Generation [97.22539956688443]
  GPT-2 generates sentences that are remarkably human-like, longer documents can ramble and do not follow human-like writing structure. We study the problem of imposing structure on long-range text. We develop a sequential controlled text generation pipeline with generation and editing.
  arXiv  Detail & Related papers  (2023-01-05T21:23:51Z)
• FAST: Improving Controllability for Text Generation with Feedback Aware Self-Training [25.75982440355576]
  Controllable text generation systems often leverage control codes to direct various properties of the output like style and length. Inspired by recent work on causal inference for NLP, this paper reveals a previously overlooked flaw in these control code-based conditional text generation algorithms. We propose two simple techniques to reduce these correlations in training sets.
  arXiv  Detail & Related papers  (2022-10-06T19:00:51Z)
• Composable Text Controls in Latent Space with ODEs [97.12426987887021]
  This paper proposes a new efficient approach for composable text operations in the compact latent space of text. By connecting pretrained LMs to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. Experiments show that composing those operators within our approach manages to generate or edit high-quality text.
  arXiv  Detail & Related papers  (2022-08-01T06:51:45Z)
• Discourse-Aware Prompt Design for Text Generation [13.835916386769474]
  We show that prompt based conditional text generation can be improved with simple and efficient methods. First, we show that a higher-level discourse structure of human written text can be modelled with textithierarchical blocking on prefix parameters. Second, we propose sparse prefix tuning by introducing textitattention sparsity on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function.
  arXiv  Detail & Related papers  (2021-12-10T18:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.