Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols

• URL: http://arxiv.org/abs/2501.13284v1
• Date: Thu, 23 Jan 2025 00:20:38 GMT
• Title: Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols
• Authors: John Joon Young Chung, Melissa Roemmele, Max Kreminski,
• Abstract summary: We introduce Toyteller, an AI-powered storytelling system where users generate a mix of story text and visuals by manipulating character symbols like they are toy-playing.
• Score: 8.676354389016101
• License:
• Abstract: We introduce Toyteller, an AI-powered storytelling system where users generate a mix of story text and visuals by directly manipulating character symbols like they are toy-playing. Anthropomorphized symbol motions can convey rich and nuanced social interactions; Toyteller leverages these motions (1) to let users steer story text generation and (2) as a visual output format that accompanies story text. We enabled motion-steered text generation and text-steered motion generation by mapping motions and text onto a shared semantic space so that large language models and motion generation models can use it as a translational layer. Technical evaluations showed that Toyteller outperforms a competitive baseline, GPT-4o. Our user study identified that toy-playing helps express intentions difficult to verbalize. However, only motions could not express all user intentions, suggesting combining it with other modalities like language. We discuss the design space of toy-playing interactions and implications for technical HCI research on human-AI interaction.

Related papers

• The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion [46.01825432018138]
  We propose a novel framework that unifies verbal and non-verbal language using multimodal language models. Our model achieves state-of-the-art performance on co-speech gesture generation. We believe unifying the verbal and non-verbal language of human motion is essential for real-world applications.
  arXiv  Detail & Related papers  (2024-12-13T19:33:48Z)
• The Art of Storytelling: Multi-Agent Generative AI for Dynamic Multimodal Narratives [3.5001789247699535]
  This paper introduces the concept of an education tool that utilizes Generative Artificial Intelligence (GenAI) to enhance storytelling for children. The system combines GenAI-driven narrative co-creation, text-to-speech conversion, and text-to-video generation to produce an engaging experience for learners.
  arXiv  Detail & Related papers  (2024-09-17T15:10:23Z)
• Generating Human Motion in 3D Scenes from Text Descriptions [60.04976442328767]
  This paper focuses on the task of generating human motions in 3D indoor scenes given text descriptions of the human-scene interactions. We propose a new approach that decomposes the complex problem into two more manageable sub-problems. For language grounding of the target object, we leverage the power of large language models; for motion generation, we design an object-centric scene representation.
  arXiv  Detail & Related papers  (2024-05-13T14:30:12Z)
• Generating Human Interaction Motions in Scenes with Text Control [66.74298145999909]
  We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model. To facilitate training, we embed annotated navigation and interaction motions within scenes.
  arXiv  Detail & Related papers  (2024-04-16T16:04:38Z)
• LivePhoto: Real Image Animation with Text-guided Motion Control [51.31418077586208]
  This work presents a practical system, named LivePhoto, which allows users to animate an image of their interest with text descriptions. We first establish a strong baseline that helps a well-learned text-to-image generator (i.e., Stable Diffusion) take an image as a further input. We then equip the improved generator with a motion module for temporal modeling and propose a carefully designed training pipeline to better link texts and motions.
  arXiv  Detail & Related papers  (2023-12-05T17:59:52Z)
• Story-to-Motion: Synthesizing Infinite and Controllable Character Animation from Long Text [14.473103773197838]
  A new task, Story-to-Motion, arises when characters are required to perform specific motions based on a long text description. Previous works in character control and text-to-motion have addressed related aspects, yet a comprehensive solution remains elusive. We propose a novel system that generates controllable, infinitely long motions and trajectories aligned with the input text.
  arXiv  Detail & Related papers  (2023-11-13T16:22:38Z)
• IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions [69.95820880360345]
  We present the first framework to synthesize the full-body motion of virtual human characters with 3D objects placed within their reach. Our system takes as input textual instructions specifying the objects and the associated intentions of the virtual characters. We show that our synthesized full-body motions appear more realistic to the participants in more than 80% of scenarios.
  arXiv  Detail & Related papers  (2022-12-14T23:59:24Z)
• TEMOS: Generating diverse human motions from textual descriptions [53.85978336198444]
  We address the problem of generating diverse 3D human motions from textual descriptions. We propose TEMOS, a text-conditioned generative model leveraging variational autoencoder (VAE) training with human motion data. We show that TEMOS framework can produce both skeleton-based animations as in prior work, as well more expressive SMPL body motions.
  arXiv  Detail & Related papers  (2022-04-25T14:53:06Z)
• Triangular Character Animation Sampling with Motion, Emotion, and Relation [78.80083186208712]
  We present a novel framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations. Our method can provide animators with an automatic way to generate 3D character animations, help synthesize interactions between Non-Player Characters (NPCs) and enhance machine emotion intelligence in virtual reality (VR)
  arXiv  Detail & Related papers  (2022-03-09T18:19:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.