Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models

• URL: http://arxiv.org/abs/2110.02891v1
• Date: Wed, 6 Oct 2021 16:17:57 GMT
• Title: Style Equalization: Unsupervised Learning of Controllable Generative Sequence Models
• Authors: Jen-Hao Rick Chang, Ashish Shrivastava, Hema Swetha Koppula, Xiaoshuai Zhang, Oncel Tuzel
• Abstract summary: We tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data.
• Score: 23.649790871960644
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Controllable generative sequence models with the capability to extract and replicate the style of specific examples enable many applications, including narrating audiobooks in different voices, auto-completing and auto-correcting written handwriting, and generating missing training samples for downstream recognition tasks. However, typical training algorithms for these controllable sequence generative models suffer from the training-inference mismatch, where the same sample is used as content and style input during training but different samples are given during inference. In this paper, we tackle the training-inference mismatch encountered during unsupervised learning of controllable generative sequence models. By introducing a style transformation module that we call style equalization, we enable training using different content and style samples and thereby mitigate the training-inference mismatch. To demonstrate its generality, we applied style equalization to text-to-speech and text-to-handwriting synthesis on three datasets. Our models achieve state-of-the-art style replication with a similar mean style opinion score as the real data. Moreover, the proposed method enables style interpolation between sequences and generates novel styles.

Related papers

• Bridging the Training-Inference Gap in LLMs by Leveraging Self-Generated Tokens [31.568675300434816]
  Language models are often trained to maximize the likelihood of the next token given past tokens in the training dataset. During inference time, they are utilized differently, generating text sequentially and auto-regressively by using previously generated tokens as input to predict the next one. This paper proposes two simple approaches based on model own generation to address this discrepancy between the training and inference time.
  arXiv  Detail & Related papers  (2024-10-18T17:48:27Z)
• Pre-Trained Vision-Language Models as Partial Annotators [40.89255396643592]
  Pre-trained vision-language models learn massive data to model unified representations of images and natural languages. In this paper, we investigate a novel "pre-trained annotating - weakly-supervised learning" paradigm for pre-trained model application and experiment on image classification tasks.
  arXiv  Detail & Related papers  (2024-05-23T17:17:27Z)
• Visual Captioning at Will: Describing Images and Videos Guided by a Few Stylized Sentences [49.66987347397398]
  Few-Shot Stylized Visual Captioning aims to generate captions in any desired style, using only a few examples as guidance during inference. We propose a framework called FS-StyleCap for this task, which utilizes a conditional encoder-decoder language model and a visual projection module.
  arXiv  Detail & Related papers  (2023-07-31T04:26:01Z)
• Prompt-Based Editing for Text Style Transfer [25.863546922455498]
  We present a prompt-based editing approach for text style transfer. We transform a prompt-based generation problem into a classification one, which is a training-free process. Our approach largely outperforms the state-of-the-art systems that have 20 times more parameters.
  arXiv  Detail & Related papers  (2023-01-27T21:31:14Z)
• Generating More Pertinent Captions by Leveraging Semantics and Style on Multi-Source Datasets [56.018551958004814]
  This paper addresses the task of generating fluent descriptions by training on a non-uniform combination of data sources. Large-scale datasets with noisy image-text pairs provide a sub-optimal source of supervision. We propose to leverage and separate semantics and descriptive style through the incorporation of a style token and keywords extracted through a retrieval component.
  arXiv  Detail & Related papers  (2021-11-24T19:00:05Z)
• Inference Time Style Control for Summarization [6.017006996402699]
  We present two novel methods that can be deployed during summary decoding on any pre-trained Transformer-based summarization model. In experiments of summarizing with simplicity control, automatic evaluation and human judges both find our models producing outputs in simpler languages while still informative.
  arXiv  Detail & Related papers  (2021-04-05T00:27:18Z)
• Diverse Semantic Image Synthesis via Probability Distribution Modeling [103.88931623488088]
  We propose a novel diverse semantic image synthesis framework. Our method can achieve superior diversity and comparable quality compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-11T18:59:25Z)
• A Correspondence Variational Autoencoder for Unsupervised Acoustic Word Embeddings [50.524054820564395]
  We propose a new unsupervised model for mapping a variable-duration speech segment to a fixed-dimensional representation. The resulting acoustic word embeddings can form the basis of search, discovery, and indexing systems for low- and zero-resource languages.
  arXiv  Detail & Related papers  (2020-12-03T19:24:42Z)
• Exploring Contextual Word-level Style Relevance for Unsupervised Style Transfer [60.07283363509065]
  Unsupervised style transfer aims to change the style of an input sentence while preserving its original content. We propose a novel attentional sequence-to-sequence model that exploits the relevance of each output word to the target style. Experimental results show that our proposed model achieves state-of-the-art performance in terms of both transfer accuracy and content preservation.
  arXiv  Detail & Related papers  (2020-05-05T10:24:28Z)
• Vector Quantized Contrastive Predictive Coding for Template-based Music Generation [0.0]
  We propose a flexible method for generating variations of discrete sequences in which tokens can be grouped into basic units. We show how these compressed representations can be used to generate variations of a template sequence by using an appropriate attention pattern in the Transformer architecture.
  arXiv  Detail & Related papers  (2020-04-21T15:58:17Z)
• Separating Content from Style Using Adversarial Learning for Recognizing Text in the Wild [103.51604161298512]
  We propose an adversarial learning framework for the generation and recognition of multiple characters in an image. Our framework can be integrated into recent recognition methods to achieve new state-of-the-art recognition accuracy.
  arXiv  Detail & Related papers  (2020-01-13T12:41:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.