CPST: Comprehension-Preserving Style Transfer for Multi-Modal Narratives

• URL: http://arxiv.org/abs/2312.08695v1
• Date: Thu, 14 Dec 2023 07:26:18 GMT
• Title: CPST: Comprehension-Preserving Style Transfer for Multi-Modal Narratives
• Authors: Yi-Chun Chen, Arnav Jhala
• Abstract summary: Among static visual narratives such as comics and manga, there are distinct visual styles in terms of presentation. The layout of both text and media elements is also significant in terms of narrative communication. We introduce the notion of comprehension-preserving style transfer (CPST) in such multi-modal domains.
• Score: 1.320904960556043
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate the challenges of style transfer in multi-modal visual narratives. Among static visual narratives such as comics and manga, there are distinct visual styles in terms of presentation. They include style features across multiple dimensions, such as panel layout, size, shape, and color. They include both visual and text media elements. The layout of both text and media elements is also significant in terms of narrative communication. The sequential transitions between panels are where readers make inferences about the narrative world. These feature differences provide an interesting challenge for style transfer in which there are distinctions between the processing of features for each modality. We introduce the notion of comprehension-preserving style transfer (CPST) in such multi-modal domains. CPST requires not only traditional metrics of style transfer but also metrics of narrative comprehension. To spur further research in this area, we present an annotated dataset of comics and manga and an initial set of algorithms that utilize separate style transfer modules for the visual, textual, and layout parameters. To test whether the style transfer preserves narrative semantics, we evaluate this algorithm through visual story cloze tests inspired by work in computational cognition of narrative systems. Understanding the connection between style and narrative semantics provides insight for applications ranging from informational brochure designs to data storytelling.

Related papers

• Panel Transitions for Genre Analysis in Visual Narratives [1.320904960556043]
  We present a novel approach to do a multi-modal analysis of genre based on comics and manga-style visual narratives. We highlight some of the limitations and challenges of our existing computational approaches in modeling subjective labels.
  arXiv  Detail & Related papers  (2023-12-14T08:05:09Z)
• A Customizable Generator for Comic-Style Visual Narrative [1.320904960556043]
  We present a theory-inspired visual narrative generator that incorporates comic-authoring idioms. The generator creates comics through sequential decision-making across layers from panel composition, object positions, panel transitions, and narrative elements.
  arXiv  Detail & Related papers  (2023-12-14T03:46:30Z)
• Don't lose the message while paraphrasing: A study on content preserving style transfer [61.38460184163704]
  Content preservation is critical for real-world applications of style transfer studies. We compare various style transfer models on the example of the formality transfer domain. We conduct a precise comparative study of several state-of-the-art techniques for style transfer.
  arXiv  Detail & Related papers  (2023-08-17T15:41:08Z)
• Text-Only Training for Visual Storytelling [107.19873669536523]
  We formulate visual storytelling as a visual-conditioned story generation problem. We propose a text-only training method that separates the learning of cross-modality alignment and story generation.
  arXiv  Detail & Related papers  (2023-08-17T09:32:17Z)
• Conversation Style Transfer using Few-Shot Learning [56.43383396058639]
  In this paper, we introduce conversation style transfer as a few-shot learning problem. We propose a novel in-context learning approach to solve the task with style-free dialogues as a pivot. We show that conversation style transfer can also benefit downstream tasks.
  arXiv  Detail & Related papers  (2023-02-16T15:27:00Z)
• Learning to Model Multimodal Semantic Alignment for Story Visualization [58.16484259508973]
  Story visualization aims to generate a sequence of images to narrate each sentence in a multi-sentence story. Current works face the problem of semantic misalignment because of their fixed architecture and diversity of input modalities. We explore the semantic alignment between text and image representations by learning to match their semantic levels in the GAN-based generative model.
  arXiv  Detail & Related papers  (2022-11-14T11:41:44Z)
• StoryTrans: Non-Parallel Story Author-Style Transfer with Discourse Representations and Content Enhancing [73.81778485157234]
  Long texts usually involve more complicated author linguistic preferences such as discourse structures than sentences. We formulate the task of non-parallel story author-style transfer, which requires transferring an input story into a specified author style. We use an additional training objective to disentangle stylistic features from the learned discourse representation to prevent the model from degenerating to an auto-encoder.
  arXiv  Detail & Related papers  (2022-08-29T08:47:49Z)
• From Theories on Styles to their Transfer in Text: Bridging the Gap with a Hierarchical Survey [10.822011920177408]
  Style transfer aims at re-writing existing texts and creating paraphrases that exhibit desired stylistic attributes. A handful of surveys give a methodological overview of the field, but they do not support researchers to focus on specific styles. We organize them into a hierarchy, highlighting the challenges for the definition of each of them, and pointing out gaps in the current research landscape.
  arXiv  Detail & Related papers  (2021-10-29T15:53:06Z)
• Improving Disentangled Text Representation Learning with Information-Theoretic Guidance [99.68851329919858]
  discrete nature of natural language makes disentangling of textual representations more challenging. Inspired by information theory, we propose a novel method that effectively manifests disentangled representations of text. Experiments on both conditional text generation and text-style transfer demonstrate the high quality of our disentangled representation.
  arXiv  Detail & Related papers  (2020-06-01T03:36:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.