Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs

• URL: http://arxiv.org/abs/2402.11218v2
• Date: Fri, 24 May 2024 04:56:41 GMT
• Title: Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs
• Authors: Xun Liang, Hanyu Wang, Shichao Song, Mengting Hu, Xunzhi Wang, Zhiyu Li, Feiyu Xiong, Bo Tang,
• Abstract summary: Controlled Text Generation (CTG) aims to produce texts that exhibit specific desired attributes. In this study, we introduce a pluggable CTG framework for Large Language Models named Dynamic Attribute Graphs-based controlled text generation.
• Score: 17.83438240818392
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Controlled Text Generation (CTG) aims to produce texts that exhibit specific desired attributes. In this study, we introduce a pluggable CTG framework for Large Language Models (LLMs) named Dynamic Attribute Graphs-based controlled text generation (DATG). This framework utilizes an attribute scorer to evaluate the attributes of sentences generated by LLMs and constructs dynamic attribute graphs. DATG modulates the occurrence of key attribute words and key anti-attribute words, achieving effective attribute control without compromising the original capabilities of the model. We conduct experiments across four datasets in two tasks: toxicity mitigation and sentiment transformation, employing five LLMs as foundational models. Our findings highlight a remarkable enhancement in control accuracy, achieving a peak improvement of 19.29% over baseline methods in the most favorable task across four datasets. Additionally, we observe a significant decrease in perplexity, markedly improving text fluency.

Related papers

• SKDU at De-Factify 4.0: Natural Language Features for AI-Generated Text-Detection [2.006175707670159]
  NELA features significantly outperform RAIDAR features in distinguishing human-written and AI-generated text. XGBoost emerged as the most effective, leveraging the rich feature sets to achieve high accuracy and generalisation.
  arXiv  Detail & Related papers  (2025-03-28T11:25:05Z)
• Towards Lightweight, Adaptive and Attribute-Aware Multi-Aspect Controllable Text Generation with Large Language Models [40.54453001537357]
  Multi-aspect controllable text generation aims to control text generation in attributes from multiple aspects. Supervised fine-tuning methods are often employed for this task due to their simplicity and effectiveness. We propose a lightweight, adaptive and attribute-aware framework for multi-aspect controllable text generation.
  arXiv  Detail & Related papers  (2025-02-19T06:56:02Z)
• Detecting Document-level Paraphrased Machine Generated Content: Mimicking Human Writing Style and Involving Discourse Features [57.34477506004105]
  Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation. We introduce novel methodologies and datasets to overcome these challenges. We propose MhBART, an encoder-decoder model designed to emulate human writing style. We also propose DTransformer, a model that integrates discourse analysis through PDTB preprocessing to encode structural features.
  arXiv  Detail & Related papers  (2024-12-17T08:47:41Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs [28.340416573162898]
  Dynamic text-attributed graphs (DyTAGs) are prevalent in various real-world scenarios. Despite their broad applicability, there is a notable scarcity of benchmark datasets tailored to DyTAGs. We introduce Dynamic Text-attributed Graph Benchmark (DTGB), a collection of large-scale, time-evolving graphs.
  arXiv  Detail & Related papers  (2024-06-17T20:16:12Z)
• Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs [36.89780636600556]
  Large language models (LLMs) have revolutionized text generation. We propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity.
  arXiv  Detail & Related papers  (2024-06-06T19:35:51Z)
• Exploiting Contextual Target Attributes for Target Sentiment Classification [53.30511968323911]
  Existing PTLM-based models for TSC can be categorized into two groups: 1) fine-tuning-based models that adopt PTLM as the context encoder; 2) prompting-based models that transfer the classification task to the text/word generation task. We present a new perspective of leveraging PTLM for TSC: simultaneously leveraging the merits of both language modeling and explicit target-context interactions via contextual target attributes.
  arXiv  Detail & Related papers  (2023-12-21T11:45:28Z)
• Successor Features for Efficient Multisubject Controlled Text Generation [48.37713738712319]
  We introduce SF-GEN, which is grounded in two primary concepts: successor features (SFs) and language model rectification. SF-GEN seamlessly integrates the two to enable dynamic steering of text generation with no need to alter the LLM's parameters. To the best of our knowledge, our research represents the first application of successor features in text generation.
  arXiv  Detail & Related papers  (2023-11-03T00:17:08Z)
• ExtractGPT: Exploring the Potential of Large Language Models for Product Attribute Value Extraction [52.14681890859275]
  E-commerce platforms require structured product data in the form of attribute-value pairs. BERT-based extraction methods require large amounts of task-specific training data. This paper explores using large language models (LLMs) as a more training-data efficient and robust alternative.
  arXiv  Detail & Related papers  (2023-10-19T07:39:00Z)
• Empower Text-Attributed Graphs Learning with Large Language Models (LLMs) [5.920353954082262]
  We propose a plug-and-play approach to empower text-attributed graphs through node generation using Large Language Models (LLMs) We employ an edge predictor to capture the structural information inherent in the raw dataset and integrate the newly generated samples into the original graph. Experiments demonstrate the outstanding performance of our proposed paradigm, particularly in low-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-15T16:04:28Z)
• Evaluating Generative Models for Graph-to-Text Generation [0.0]
  We explore the capability of generative models to generate descriptive text from graph data in a zero-shot setting. Our results demonstrate that generative models are capable of generating fluent and coherent text. However, our error analysis reveals that generative models still struggle with understanding the semantic relations between entities.
  arXiv  Detail & Related papers  (2023-07-27T09:03:05Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Controlled Text Generation with Hidden Representation Transformations [12.576140288264835]
  CHRT steers large language models to generate text pertaining to certain attributes (such as toxicity) We employ a contrastive-learning framework to learn these transformations that can be combined to gain multi-attribute control. CHRT outperforms all the baselines in the task of detoxification, positive sentiment steering, and text simplification.
  arXiv  Detail & Related papers  (2023-05-30T17:21:17Z)
• Investigating Pretrained Language Models for Graph-to-Text Generation [55.55151069694146]
  Graph-to-text generation aims to generate fluent texts from graph-based data. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further.
  arXiv  Detail & Related papers  (2020-07-16T16:05:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.