BotDirector: Robot Storytelling Across the Symmetrical Reality with Multi-modal Interactions

• URL: http://arxiv.org/abs/2606.03223v1
• Date: Tue, 02 Jun 2026 06:33:35 GMT
• Title: BotDirector: Robot Storytelling Across the Symmetrical Reality with Multi-modal Interactions
• Abstract summary: We propose an interactive system that facilitates robot storytelling with tangible and natural language interactions.<n>Children arrange the playground with their own stuff and create narratives with an LLM agent.<n>The created narratives are transformed into a motion sequence based on the map and characters, and the motions are executed by self-navigating swarm robots.
• Score: 25.188575753340416
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Robot storytelling offers a unique blend of technological innovation and creative expression that engages children in unprecedented ways. However, the technical aspects are often too complicated for children. We propose an interactive system that facilitates robot storytelling with tangible and natural language interactions. Children arrange the playground with their own stuff and create narratives with an LLM agent. The created narratives are transformed into a motion sequence based on the map and characters, and the motions are executed by self-navigating swarm robots. This system enhances robot storytelling with flexible scenarios, enabling young children to create robot dramas with everyday objects.

Related papers

• LLM-Powered Interactive Robotic Action Synthesis from Multimodal Speech, Gestures, and Music [1.7188280334580195]
  This paper introduces a novel framework that synthesizes complex robotic actions from a rich tapestry of multimodal human inputs.<n>Our system architecture integrates a speech transcription model, a gesture recognition module, and a signal processing pipeline for beat detection.<n>The framework has ability to interpret and fuse semantic commands from speech, deictic information from gestures, and rhythmic cues from music.
  arXiv  Detail & Related papers  (2026-06-30T05:35:26Z)
• RoboTales: ROBOTic Anthropomorphic LEarning Systems [0.18501087920760131]
  RoboTales is a low-cost robotic storytelling system that animates narratives using expressive sock puppetry.<n>It synchronizes narration, gestures, and mouth movements to perform character-driven stories.<n>In a pilot study, puppet-based storytelling outperformed a gesture-only mode, producing higher HRIES ratings and improved story recall.
  arXiv  Detail & Related papers  (2026-06-24T17:41:02Z)
• Aesthetics of Robot-Mediated Applied Drama: A Case Study on REMind [3.8229109732721778]
  Social robots are increasingly used in education, but most applications cast them as tutors offering explanation-based instruction.<n>We explore an alternative: Robot-Mediated Applied Drama (RMAD), in which robots function as life-like puppets in interactive dramatic experiences.<n>This paper presents REMind, an anti-bullying robot role-play game that helps children rehearse bystander intervention and peer support.
  arXiv  Detail & Related papers  (2026-03-25T00:58:01Z)
• Tinker Tales: Supporting Child-AI Collaboration through Co-Creative Storytelling with Educational Scaffolding [7.821369295879863]
  Tinker Tales is a storytelling system designed with narrative and social-emotional scaffolding to support child-AI collaboration.<n>Children shape and refine stories by placing and moving story elements and interacting with the AI through tangible and voice-based interaction.
  arXiv  Detail & Related papers  (2026-02-04T00:48:12Z)
• H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos [58.006918399913665]
  We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos.<n>Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale.<n>At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions.
  arXiv  Detail & Related papers  (2025-12-10T07:59:45Z)
• RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video [56.9581053843815]
  We introduce RobotSeg, a foundation model for robot segmentation in image and video.<n>It addresses the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations.<n>It achieves state-of-the-art performance on both images and videos.
  arXiv  Detail & Related papers  (2025-11-28T07:51:02Z)
• INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLM [9.217332197059001]
  In this work, we propose INTENTION, a novel framework enabling robots with learned interactive intuition and autonomous manipulation in diverse scenarios.<n>We introduce Memory Graph to record scenes from previous task interactions which embodies human-like understanding and decision-making about different tasks in real world.<n>Meanwhile, we design an Intuitive Perceptor that extracts physical relations and affordances from visual scenes.
  arXiv  Detail & Related papers  (2025-08-06T23:27:22Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)
• Controlling diverse robots by inferring Jacobian fields with deep networks [48.279199537720714]
  Mirroring the complex structures and diverse functions of natural organisms is a long-standing challenge in robotics.<n>We introduce a method that uses deep neural networks to map a video stream of a robot to its visuomotor Jacobian field.<n>Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot.
  arXiv  Detail & Related papers  (2024-07-11T17:55:49Z)
• Learning an Actionable Discrete Diffusion Policy via Large-Scale Actionless Video Pre-Training [69.54948297520612]
  Learning a generalist embodied agent poses challenges, primarily stemming from the scarcity of action-labeled robotic datasets. We introduce a novel framework to tackle these challenges, which leverages a unified discrete diffusion to combine generative pre-training on human videos and policy fine-tuning on a small number of action-labeled robot videos. Our method generates high-fidelity future videos for planning and enhances the fine-tuned policies compared to previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-02-22T09:48:47Z)
• Synthesis and Execution of Communicative Robotic Movements with Generative Adversarial Networks [59.098560311521034]
  We focus on how to transfer on two different robotic platforms the same kinematics modulation that humans adopt when manipulating delicate objects. We choose to modulate the velocity profile adopted by the robots' end-effector, inspired by what humans do when transporting objects with different characteristics. We exploit a novel Generative Adversarial Network architecture, trained with human kinematics examples, to generalize over them and generate new and meaningful velocity profiles.
  arXiv  Detail & Related papers  (2022-03-29T15:03:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.