Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs

• URL: http://arxiv.org/abs/2406.04460v1
• Date: Thu, 6 Jun 2024 19:35:51 GMT
• Title: Evaluating the Smooth Control of Attribute Intensity in Text Generation with LLMs
• Authors: Shang Zhou, Feng Yao, Chengyu Dong, Zihan Wang, Jingbo Shang,
• Abstract summary: Large language models (LLMs) have revolutionized text generation. We propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity.
• Score: 36.89780636600556
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Controlling the attribute intensity of text generation is crucial across scenarios (e.g., writing conciseness, chatting emotion, and explanation clarity). The remarkable capabilities of large language models (LLMs) have revolutionized text generation, prompting us to explore such \emph{smooth control} of LLM generation. Specifically, we propose metrics to assess the range, calibration, and consistency of the generated text's attribute intensity in response to varying control values, as well as its relevance to the intended context. To quantify the attribute intensity and context relevance, we propose an effective evaluation framework leveraging the Elo rating system and GPT4, both renowned for their robust alignment with human judgment. We look into two viable training-free methods for achieving smooth control of LLMs: (1) Prompting with semantic shifters, and (2) Modifying internal model representations. The evaluations of these two methods are conducted on $5$ different attributes with various models. Our code and dataset can be obtained from \url{https://github.com/ShangDataLab/Smooth-Control}.

Related papers

• Feature-Level Insights into Artificial Text Detection with Sparse Autoencoders [20.557610461777344]
  We use Sparse Autoencoders (SAE) to extract features from Gemma-2-2b residual stream. We identify both interpretable and efficient features, analyzing their semantics and relevance. Our methods offer valuable insights into how texts from various models differ from human-written content.
  arXiv  Detail & Related papers  (2025-03-05T15:33:52Z)
• Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore [51.65730053591696]
  We propose a simple yet effective black-box zero-shot detection approach based on the observation that human-written texts typically contain more grammatical errors than LLM-generated texts. Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods.
  arXiv  Detail & Related papers  (2024-05-07T12:57:01Z)
• Controlled Text Generation for Large Language Model with Dynamic Attribute Graphs [17.83438240818392]
  Controlled Text Generation (CTG) aims to produce texts that exhibit specific desired attributes. In this study, we introduce a pluggable CTG framework for Large Language Models named Dynamic Attribute Graphs-based controlled text generation.
  arXiv  Detail & Related papers  (2024-02-17T08:14:37Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Harnessing the Plug-and-Play Controller by Prompting [12.705251690623495]
  This paper introduces a novel method for flexible attribute control in text generation using pre-trained language models (PLMs) The proposed approach aims to enhance the fluency of generated text by guiding the generation process with PPCs.
  arXiv  Detail & Related papers  (2024-02-06T17:18:25Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• Prompt Highlighter: Interactive Control for Multi-Modal LLMs [50.830448437285355]
  This study targets a critical aspect of multi-modal LLMs' (LLMs&VLMs) inference: explicit controllable text generation. We introduce a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. We find that, during inference, guiding the models with highlighted tokens through the attention weights leads to more desired outputs.
  arXiv  Detail & Related papers  (2023-12-07T13:53:29Z)
• Successor Features for Efficient Multisubject Controlled Text Generation [48.37713738712319]
  We introduce SF-GEN, which is grounded in two primary concepts: successor features (SFs) and language model rectification. SF-GEN seamlessly integrates the two to enable dynamic steering of text generation with no need to alter the LLM's parameters. To the best of our knowledge, our research represents the first application of successor features in text generation.
  arXiv  Detail & Related papers  (2023-11-03T00:17:08Z)
• FAST: Improving Controllability for Text Generation with Feedback Aware Self-Training [25.75982440355576]
  Controllable text generation systems often leverage control codes to direct various properties of the output like style and length. Inspired by recent work on causal inference for NLP, this paper reveals a previously overlooked flaw in these control code-based conditional text generation algorithms. We propose two simple techniques to reduce these correlations in training sets.
  arXiv  Detail & Related papers  (2022-10-06T19:00:51Z)
• Controllable Dialogue Generation with Disentangled Multi-grained Style Specification and Attribute Consistency Reward [47.96949534259019]
  We propose a controllable dialogue generation model to steer response generation under multi-attribute constraints. We categorize the commonly used control attributes into global and local ones, which possess different granularities of effects on response generation. Our model can significantly outperform competitive baselines in terms of response quality, content diversity and controllability.
  arXiv  Detail & Related papers  (2021-09-14T14:29:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.