Reinforcement Learning with Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation

• URL: http://arxiv.org/abs/2402.14146v1
• Date: Wed, 21 Feb 2024 22:02:37 GMT
• Title: Reinforcement Learning with Dynamic Multi-Reward Weighting for Multi-Style Controllable Generation
• Authors: Karin de Langis, Ryan Koo, Dongyeop Kang
• Abstract summary: Humans often employ multiple styles simultaneously. We investigate various formulations of multiple style rewards for a reinforcement learning approach to controlled multi-style generation. All code and data for RL pipelines with multiple style attributes will be publicly available.
• Score: 17.937198263444046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Style is an integral component of text that expresses a diverse set of information, including interpersonal dynamics (e.g. formality) and the author's emotions or attitudes (e.g. disgust). Humans often employ multiple styles simultaneously. An open question is how large language models can be explicitly controlled so that they weave together target styles when generating text: for example, to produce text that is both negative and non-toxic. Previous work investigates the controlled generation of a single style, or else controlled generation of a style and other attributes. In this paper, we expand this into controlling multiple styles simultaneously. Specifically, we investigate various formulations of multiple style rewards for a reinforcement learning (RL) approach to controlled multi-style generation. These reward formulations include calibrated outputs from discriminators and dynamic weighting by discriminator gradient magnitudes. We find that dynamic weighting generally outperforms static weighting approaches, and we explore its effectiveness in 2- and 3-style control, even compared to strong baselines like plug-and-play model. All code and data for RL pipelines with multiple style attributes will be publicly available.

Related papers

• Say Anything with Any Style [9.50806457742173]
  Say Anything withAny Style queries the discrete style representation via a generative model with a learned style codebook. Our approach surpasses state-of-theart methods in terms of both lip-synchronization and stylized expression.
  arXiv  Detail & Related papers  (2024-03-11T01:20:03Z)
• TextDiffuser-2: Unleashing the Power of Language Models for Text Rendering [118.30923824681642]
  TextDiffuser-2 aims to unleash the power of language models for text rendering. We utilize the language model within the diffusion model to encode the position and texts at the line level. We conduct extensive experiments and incorporate user studies involving human participants as well as GPT-4V.
  arXiv  Detail & Related papers  (2023-11-28T04:02:40Z)
• STEER: Unified Style Transfer with Expert Reinforcement [71.3995732115262]
  STEER: Unified Style Transfer with Expert Reinforcement, is a unified frame-work developed to overcome the challenge of limited parallel data for style transfer. We show STEER is robust, maintaining its style transfer capabilities on out-of-domain data, and surpassing nearly all baselines across various styles.
  arXiv  Detail & Related papers  (2023-11-13T09:02:30Z)
• GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latents [3.229105662984031]
  GestureDiffuCLIP is a neural network framework for synthesizing realistic, stylized co-speech gestures with flexible style control. Our system learns a latent diffusion model to generate high-quality gestures and infuses the CLIP representations of style into the generator. Our system can be extended to allow fine-grained style control of individual body parts.
  arXiv  Detail & Related papers  (2023-03-26T03:35:46Z)
• Audience-Centric Natural Language Generation via Style Infusion [5.6732899077715375]
  We propose the novel task of style infusion - infusing the stylistic preferences of audiences in pretrained language generation models. We leverage limited pairwise human judgments to bootstrap a style analysis model and augment our seed set of judgments. Our infusion approach can generate compelling stylized examples with generic text prompts.
  arXiv  Detail & Related papers  (2023-01-24T19:57:50Z)
• Unsupervised Neural Stylistic Text Generation using Transfer learning and Adapters [66.17039929803933]
  We propose a novel transfer learning framework which updates only $0.3%$ of model parameters to learn style specific attributes for response generation. We learn style specific attributes from the PERSONALITY-CAPTIONS dataset.
  arXiv  Detail & Related papers  (2022-10-07T00:09:22Z)
• Twist Decoding: Diverse Generators Guide Each Other [116.20780037268801]
  We introduce Twist decoding, a simple and general inference algorithm that generates text while benefiting from diverse models. Our method does not assume the vocabulary, tokenization or even generation order is shared.
  arXiv  Detail & Related papers  (2022-05-19T01:27:53Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• DyStyle: Dynamic Neural Network for Multi-Attribute-Conditioned Style Editing [12.80013698957431]
  A Dynamic Style Manipulation Network (DyStyle) is proposed to perform attribute-conditioned style editing. A novel easy-to-hard training procedure is introduced for efficient and stable training of the DyStyle network. Our approach demonstrates fine-grained disentangled edits along multiple numeric and binary attributes.
  arXiv  Detail & Related papers  (2021-09-22T13:50:51Z)
• Multi-type Disentanglement without Adversarial Training [48.51678740102892]
  Controlling the style of natural language by disentangling the latent space is an important step towards interpretable machine learning. We propose a unified distribution-controlling method, which provides each specific style value with a unique representation. We also propose multiple loss functions to achieve a style-content disentanglement as well as a disentanglement among multiple style types.
  arXiv  Detail & Related papers  (2020-12-16T11:47:18Z)
• Multi-Style Transfer with Discriminative Feedback on Disjoint Corpus [9.793194158416854]
  Style transfer has been widely explored in natural language generation with non-parallel corpus. A common shortcoming of existing approaches is the prerequisite of joint annotations across all the stylistic dimensions. We show the ability of our model to control styles across multiple style dimensions while preserving content of the input text.
  arXiv  Detail & Related papers  (2020-10-22T10:16:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.