On Text Style Transfer via Style Masked Language Models

• URL: http://arxiv.org/abs/2210.06394v1
• Date: Wed, 12 Oct 2022 16:44:06 GMT
• Title: On Text Style Transfer via Style Masked Language Models
• Authors: Sharan Narasimhan, Pooja Shekar, Suvodip Dey, Maunendra Sankar Desarkar
• Abstract summary: Text Style Transfer (TST) is performable through approaches such as latent space disentanglement, cycleconsistency losses, prototype editing. We present a prototype editing approach, which involves two key phases a) Masking of source style-associated tokens and b) Reconstruction of this source-style masked sentence conditioned with the target style. We empirically show that this non-generational approach well suites the "content preserving" criteria for a task like TST, even for a complex baseline like Discourse.
• Score: 5.754152248672319
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text Style Transfer (TST) is performable through approaches such as latent space disentanglement, cycle-consistency losses, prototype editing etc. The prototype editing approach, which is known to be quite successful in TST, involves two key phases a) Masking of source style-associated tokens and b) Reconstruction of this source-style masked sentence conditioned with the target style. We follow a similar transduction method, in which we transpose the more difficult direct source to target TST task to a simpler Style-Masked Language Model (SMLM) Task, wherein, similar to BERT \cite{bert}, the goal of our model is now to reconstruct the source sentence from its style-masked version. We arrive at the SMLM mechanism naturally by formulating prototype editing/ transduction methods in a probabilistic framework, where TST resolves into estimating a hypothetical parallel dataset from a partially observed parallel dataset, wherein each domain is assumed to have a common latent style-masked prior. To generate this style-masked prior, we use "Explainable Attention" as our choice of attribution for a more precise style-masking step and also introduce a cost-effective and accurate "Attribution-Surplus" method of determining the position of masks from any arbitrary attribution model in O(1) time. We empirically show that this non-generational approach well suites the "content preserving" criteria for a task like TST, even for a complex style like Discourse Manipulation. Our model, the Style MLM, outperforms strong TST baselines and is on par with state-of-the-art TST models, which use complex architectures and orders of more parameters.

Related papers

• Style-Specific Neurons for Steering LLMs in Text Style Transfer [55.06697862691798]
  Text style transfer (TST) aims to modify the style of a text without altering its original meaning. We present sNeuron-TST, a novel approach for steering large language models using style-specific neurons.
  arXiv  Detail & Related papers  (2024-10-01T11:25:36Z)
• MaskInversion: Localized Embeddings via Optimization of Explainability Maps [49.50785637749757]
  MaskInversion generates a context-aware embedding for a query image region specified by a mask at test time. It can be used for a broad range of tasks, including open-vocabulary class retrieval, referring expression comprehension, as well as for localized captioning and image generation.
  arXiv  Detail & Related papers  (2024-07-29T14:21:07Z)
• Unsupervised Text Style Transfer via LLMs and Attention Masking with Multi-way Interactions [18.64326057581588]
  Unsupervised Text Style Transfer (UTST) has emerged as a critical task within the domain of Natural Language Processing (NLP) We propose four ways of interactions, that are pipeline framework with tuned orders; knowledge distillation from Large Language Models (LLMs) to attention masking model; in-context learning with constructed parallel examples. We empirically show these multi-way interactions can improve the baselines in certain perspective of style strength, content preservation and text fluency.
  arXiv  Detail & Related papers  (2024-02-21T09:28:02Z)
• FLIP: Fine-grained Alignment between ID-based Models and Pretrained Language Models for CTR Prediction [49.510163437116645]
  Click-through rate (CTR) prediction plays as a core function module in personalized online services. Traditional ID-based models for CTR prediction take as inputs the one-hot encoded ID features of tabular modality. Pretrained Language Models(PLMs) has given rise to another paradigm, which takes as inputs the sentences of textual modality. We propose to conduct Fine-grained feature-level ALignment between ID-based Models and Pretrained Language Models(FLIP) for CTR prediction.
  arXiv  Detail & Related papers  (2023-10-30T11:25:03Z)
• Stylized Data-to-Text Generation: A Case Study in the E-Commerce Domain [53.22419717434372]
  We propose a new task, namely stylized data-to-text generation, whose aim is to generate coherent text according to a specific style. This task is non-trivial, due to three challenges: the logic of the generated text, unstructured style reference, and biased training samples. We propose a novel stylized data-to-text generation model, named StyleD2T, comprising three components: logic planning-enhanced data embedding, mask-based style embedding, and unbiased stylized text generation.
  arXiv  Detail & Related papers  (2023-05-05T03:02:41Z)
• Typhoon: Towards an Effective Task-Specific Masking Strategy for Pre-trained Language Models [0.0]
  In this paper, we explore a task-specific masking framework for pre-trained large language models. We develop our own masking algorithm, Typhoon, based on token input gradients, and compare this with other standard baselines. Our implementation can be found in a public Github Repository.
  arXiv  Detail & Related papers  (2023-03-27T22:27:23Z)
• Replacing Language Model for Style Transfer [6.364517234783756]
  We introduce replacing language model (RLM), a sequence-to-sequence language modeling framework for text style transfer (TST) Our method autoregressively replaces each token of the source sentence with a text span that has a similar meaning but in the target style. The new span is generated via a non-autoregressive masked language model, which can better preserve the local-contextual meaning of the replaced token.
  arXiv  Detail & Related papers  (2022-11-14T13:35:55Z)
• Syntax Matters! Syntax-Controlled in Text Style Transfer [24.379552683296392]
  Existing text style transfer (TST) methods rely on style classifiers to disentangle the text's content and style attributes. We propose a novel Syntax-Aware Controllable Generation (SACG) model, which includes a syntax-aware style classifier. We show that our proposed method significantly outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-08-12T17:35:23Z)
• MST: Masked Self-Supervised Transformer for Visual Representation [52.099722121603506]
  Transformer has been widely used for self-supervised pre-training in Natural Language Processing (NLP) We present a novel Masked Self-supervised Transformer approach named MST, which can explicitly capture the local context of an image. MST achieves Top-1 accuracy of 76.9% with DeiT-S only using 300-epoch pre-training by linear evaluation.
  arXiv  Detail & Related papers  (2021-06-10T11:05:18Z)
• Generative Pre-training for Paraphrase Generation by Representing and Predicting Spans in Exemplars [0.8411385346896411]
  This paper presents a novel approach to paraphrasing sentences, extended from the GPT-2 model. We develop a template masking technique, named first-order masking, to masked out irrelevant words in exemplars utilizing POS taggers. Our proposed approach outperforms competitive baselines, especially in the semantic preservation aspect.
  arXiv  Detail & Related papers  (2020-11-29T11:36:13Z)
• Contextual Text Style Transfer [73.66285813595616]
  Contextual Text Style Transfer aims to translate a sentence into a desired style with its surrounding context taken into account. We propose a Context-Aware Style Transfer (CAST) model, which uses two separate encoders for each input sentence and its surrounding context. Two new benchmarks, Enron-Context and Reddit-Context, are introduced for formality and offensiveness style transfer.
  arXiv  Detail & Related papers  (2020-04-30T23:01:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.