Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training

• URL: http://arxiv.org/abs/2309.10929v1
• Date: Tue, 19 Sep 2023 21:01:40 GMT
• Title: Specializing Small Language Models towards Complex Style Transfer via Latent Attribute Pre-Training
• Authors: Ruiqi Xu, Yongfeng Huang, Xin Chen, Lin Zhang
• Abstract summary: We introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact.
• Score: 29.143887057933327
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we introduce the concept of complex text style transfer tasks, and constructed complex text datasets based on two widely applicable scenarios. Our dataset is the first large-scale data set of its kind, with 700 rephrased sentences and 1,000 sentences from the game Genshin Impact. While large language models (LLM) have shown promise in complex text style transfer, they have drawbacks such as data privacy concerns, network instability, and high deployment costs. To address these issues, we explore the effectiveness of small models (less than T5-3B) with implicit style pre-training through contrastive learning. We also propose a method for automated evaluation of text generation quality based on alignment with human evaluations using ChatGPT. Finally, we compare our approach with existing methods and show that our model achieves state-of-art performances of few-shot text style transfer models.

Related papers

• A Simple yet Efficient Ensemble Approach for AI-generated Text Detection [0.5840089113969194]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in generating text that closely resembles human writing. It is essential to build automated approaches capable of distinguishing between artificially generated text and human-authored text. We propose a simple yet efficient solution by ensembling predictions from multiple constituent LLMs.
  arXiv  Detail & Related papers  (2023-11-06T13:11:02Z)
• Prefix-Tuning Based Unsupervised Text Style Transfer [29.86587278794342]
  Unsupervised text style transfer aims at training a generative model that can alter the style of the input sentence while preserving its content. In this paper, we employ powerful pre-trained large language models and present a new prefix-tuning-based method for unsupervised text style transfer.
  arXiv  Detail & Related papers  (2023-10-23T06:13:08Z)
• Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain [53.22419717434372]
  We propose a new task, namely stylized data-to-text generation, whose aim is to generate coherent text according to a specific style. This task is non-trivial, due to three challenges: the logic of the generated text, unstructured style reference, and biased training samples. We propose a novel stylized data-to-text generation model, named StyleD2T, comprising three components: logic planning-enhanced data embedding, mask-based style embedding, and unbiased stylized text generation.
  arXiv  Detail & Related papers  (2023-05-05T03:02:41Z)
• Unsupervised Neural Stylistic Text Generation using Transfer learning and Adapters [66.17039929803933]
  We propose a novel transfer learning framework which updates only $0.3%$ of model parameters to learn style specific attributes for response generation. We learn style specific attributes from the PERSONALITY-CAPTIONS dataset.
  arXiv  Detail & Related papers  (2022-10-07T00:09:22Z)
• Prompt-and-Rerank: A Method for Zero-Shot and Few-Shot Arbitrary Textual Style Transfer with Small Language Models [27.454582992694974]
  We propose a method for arbitrary textual style transfer (TST) Our method, Prompt-and-Rerank, is based on a mathematical formulation of the TST task. Empirically, our method enables small pre-trained language models to perform on par with state-of-the-art large-scale models.
  arXiv  Detail & Related papers  (2022-05-23T17:57:15Z)
• Non-Parallel Text Style Transfer with Self-Parallel Supervision [19.441780035577352]
  We propose LaMer, a novel text style transfer framework based on large-scale language models. LaMer first mines the roughly parallel expressions in the non-parallel datasets with scene graphs, and then employs MLE training, followed by imitation learning refinement, to leverage the intrinsic parallelism within the data. On two benchmark tasks (sentiment & formality transfer) and a newly proposed challenging task (political stance transfer), our model achieves qualitative advances in transfer accuracy, content preservation, and fluency.
  arXiv  Detail & Related papers  (2022-04-18T01:38:35Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Syntax-Enhanced Pre-trained Model [49.1659635460369]
  We study the problem of leveraging the syntactic structure of text to enhance pre-trained models such as BERT and RoBERTa. Existing methods utilize syntax of text either in the pre-training stage or in the fine-tuning stage, so that they suffer from discrepancy between the two stages. We present a model that utilizes the syntax of text in both pre-training and fine-tuning stages.
  arXiv  Detail & Related papers  (2020-12-28T06:48:04Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Contextual Text Style Transfer [73.66285813595616]
  Contextual Text Style Transfer aims to translate a sentence into a desired style with its surrounding context taken into account. We propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context. Two new benchmarks, Enron-Context and Reddit-Context, are introduced for formality and offensiveness style transfer.
  arXiv  Detail & Related papers  (2020-04-30T23:01:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.