Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols

• URL: http://arxiv.org/abs/2501.13284v1
• Date: Thu, 23 Jan 2025 00:20:38 GMT
• Title: Toyteller: AI-powered Visual Storytelling Through Toy-Playing with Character Symbols
• Authors: John Joon Young Chung, Melissa Roemmele, Max Kreminski,
• Abstract summary: We introduce Toyteller, an AI-powered storytelling system where users generate a mix of story text and visuals by manipulating character symbols like they are toy-playing.
• Score: 8.676354389016101
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce Toyteller, an AI-powered storytelling system where users generate a mix of story text and visuals by directly manipulating character symbols like they are toy-playing. Anthropomorphized symbol motions can convey rich and nuanced social interactions; Toyteller leverages these motions (1) to let users steer story text generation and (2) as a visual output format that accompanies story text. We enabled motion-steered text generation and text-steered motion generation by mapping motions and text onto a shared semantic space so that large language models and motion generation models can use it as a translational layer. Technical evaluations showed that Toyteller outperforms a competitive baseline, GPT-4o. Our user study identified that toy-playing helps express intentions difficult to verbalize. However, only motions could not express all user intentions, suggesting combining it with other modalities like language. We discuss the design space of toy-playing interactions and implications for technical HCI research on human-AI interaction.

Related papers

• Tinker Tales: Interactive Storytelling Framework for Early Childhood Narrative Development and AI Literacy [9.415578811438992]
  The framework integrates tangible and speech-based interactions with AI through NFC chip-attached pawns and tokens. Children select and define key story elements-such as characters, places, items, and emotions-using the pawns and tokens. For evaluation, several game sessions were simulated with a child AI agent, and the quality and safety of the generated stories were assessed.
  arXiv  Detail & Related papers  (2025-04-17T17:47:55Z)
• MoCha: Towards Movie-Grade Talking Character Synthesis [62.007000023747445]
  We introduce Talking Characters, a more realistic task to generate talking character animations directly from speech and text. Unlike talking head, Talking Characters aims at generating the full portrait of one or more characters beyond the facial region. We propose MoCha, the first of its kind to generate talking characters.
  arXiv  Detail & Related papers  (2025-03-30T04:22:09Z)
• WhatELSE: Shaping Narrative Spaces at Configurable Level of Abstraction for AI-bridged Interactive Storytelling [11.210282687859534]
  WhatELSE is an AI-bridged IN authoring system that creates narrative possibility spaces from example stories. We show that WhatELSE enables authors to perceive and edit the narrative space and generates engaging interactive narratives at play-time.
  arXiv  Detail & Related papers  (2025-02-25T21:02:15Z)
• The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion [46.01825432018138]
  We propose a novel framework that unifies verbal and non-verbal language using multimodal language models.<n>Our model achieves state-of-the-art performance on co-speech gesture generation.<n>We believe unifying the verbal and non-verbal language of human motion is essential for real-world applications.
  arXiv  Detail & Related papers  (2024-12-13T19:33:48Z)
• The Art of Storytelling: Multi-Agent Generative AI for Dynamic Multimodal Narratives [3.5001789247699535]
  This paper introduces the concept of an education tool that utilizes Generative Artificial Intelligence (GenAI) to enhance storytelling for children. The system combines GenAI-driven narrative co-creation, text-to-speech conversion, and text-to-video generation to produce an engaging experience for learners.
  arXiv  Detail & Related papers  (2024-09-17T15:10:23Z)
• Generating Human Motion in 3D Scenes from Text Descriptions [60.04976442328767]
  This paper focuses on the task of generating human motions in 3D indoor scenes given text descriptions of the human-scene interactions. We propose a new approach that decomposes the complex problem into two more manageable sub-problems. For language grounding of the target object, we leverage the power of large language models; for motion generation, we design an object-centric scene representation.
  arXiv  Detail & Related papers  (2024-05-13T14:30:12Z)
• Generating Human Interaction Motions in Scenes with Text Control [66.74298145999909]
  We present TeSMo, a method for text-controlled scene-aware motion generation based on denoising diffusion models. Our approach begins with pre-training a scene-agnostic text-to-motion diffusion model. To facilitate training, we embed annotated navigation and interaction motions within scenes.
  arXiv  Detail & Related papers  (2024-04-16T16:04:38Z)
• LivePhoto: Real Image Animation with Text-guided Motion Control [51.31418077586208]
  This work presents a practical system, named LivePhoto, which allows users to animate an image of their interest with text descriptions. We first establish a strong baseline that helps a well-learned text-to-image generator (i.e., Stable Diffusion) take an image as a further input. We then equip the improved generator with a motion module for temporal modeling and propose a carefully designed training pipeline to better link texts and motions.
  arXiv  Detail & Related papers  (2023-12-05T17:59:52Z)
• Story-to-Motion: Synthesizing Infinite and Controllable Character Animation from Long Text [14.473103773197838]
  A new task, Story-to-Motion, arises when characters are required to perform specific motions based on a long text description. Previous works in character control and text-to-motion have addressed related aspects, yet a comprehensive solution remains elusive. We propose a novel system that generates controllable, infinitely long motions and trajectories aligned with the input text.
  arXiv  Detail & Related papers  (2023-11-13T16:22:38Z)
• IMoS: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactions [69.95820880360345]
  We present the first framework to synthesize the full-body motion of virtual human characters with 3D objects placed within their reach. Our system takes as input textual instructions specifying the objects and the associated intentions of the virtual characters. We show that our synthesized full-body motions appear more realistic to the participants in more than 80% of scenarios.
  arXiv  Detail & Related papers  (2022-12-14T23:59:24Z)
• Triangular Character Animation Sampling with Motion, Emotion, and Relation [78.80083186208712]
  We present a novel framework to sample and synthesize animations by associating the characters' body motions, facial expressions, and social relations. Our method can provide animators with an automatic way to generate 3D character animations, help synthesize interactions between Non-Player Characters (NPCs) and enhance machine emotion intelligence in virtual reality (VR)
  arXiv  Detail & Related papers  (2022-03-09T18:19:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.