PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR

• URL: http://arxiv.org/abs/2603.02366v1
• Date: Mon, 02 Mar 2026 20:11:44 GMT
• Title: PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR
• Authors: Esen K. Tütüncü, Qian Zhou, Frederik Brudy, George Fitzmaurice, Fraser Anderson,
• Abstract summary: We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props.<n>A multi-agent AI pipeline interprets these actions into Intent Frames -structured narrative beats visualized as rearrangeable story marbles on a timeline.
• Score: 15.355161864831432
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current AI writing tools, which rely on text prompts, poorly support the spatial and interactive nature of storytelling where ideas emerge from direct manipulation and play. We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props. A multi-agent AI pipeline interprets these actions into Intent Frames -structured narrative beats visualized as rearrangeable story marbles on a timeline. A large language model then transforms the user's assembled sequence into a final narrative. A user study (N=13) with writers from varying domains found that PlayWrite fosters a highly improvisational and playful process. Users treated the AI as a collaborative partner, using its unexpected responses to spark new ideas and overcome creative blocks. PlayWrite demonstrates an approach for co-creative systems that move beyond text to embrace direct manipulation and play as core interaction modalities.

Related papers

• Elsewise: Authoring AI-Based Interactive Narrative with Possibility Space Visualization [23.460775222928635]
  Generative AI can enable new forms of interactive narrative (IN) by improvisationally expanding on pre-authored content.<n>We introduce Elsewise: an authoring tool for AI-based INs that implements a novel Bundled Storyline concept.<n>A user study shows that our approach improves author anticipation of player-experienced narrative, leading to more effective control and exploration of the narrative possibility spaces.
  arXiv  Detail & Related papers  (2025-12-21T02:06:03Z)
• RoboOmni: Proactive Robot Manipulation in Omni-modal Context [165.09049429566238]
  We introduce cross-modal contextual instructions, a new setting where intent is derived from spoken dialogue, environmental sounds, and visual cues rather than explicit commands.<n>We present RoboOmni, a framework based on end-to-end omni-modal LLMs that unifies intention recognition, interaction confirmation, and action execution.<n>Experiments in simulation and real-world settings show that Robo Omni surpasses text- and ASR-based baselines in success rate, inference speed, intention recognition, and proactive assistance.
  arXiv  Detail & Related papers  (2025-10-27T18:49:03Z)
• Multi-Actor Generative Artificial Intelligence as a Game Engine [49.360775442760314]
  Generative AI can be used in multi-actor environments with purposes ranging from social science modeling to interactive narrative and AI evaluation.<n>We argue here that a good approach is to take inspiration from tabletop role-playing games (TTRPGs), where a Game Master (GM) is responsible for the environment and generates all parts of the story not directly determined by the voluntary actions of player characters.
  arXiv  Detail & Related papers  (2025-07-10T22:31:09Z)
• WhatELSE: Shaping Narrative Spaces at Configurable Level of Abstraction for AI-bridged Interactive Storytelling [11.210282687859534]
  WhatELSE is an AI-bridged IN authoring system that creates narrative possibility spaces from example stories.<n>We show that WhatELSE enables authors to perceive and edit the narrative space and generates engaging interactive narratives at play-time.
  arXiv  Detail & Related papers  (2025-02-25T21:02:15Z)
• Towards Enhanced Immersion and Agency for LLM-based Interactive Drama [55.770617779283064]
  This paper begins with understanding interactive drama from two aspects: Immersion, the player's feeling of being present in the story, and Agency.<n>To enhance these two aspects, we first propose Playwriting-guided Generation, a novel method that helps LLMs craft dramatic stories with substantially improved structures and narrative quality.
  arXiv  Detail & Related papers  (2025-02-25T06:06:16Z)
• Agents' Room: Narrative Generation through Multi-step Collaboration [54.98886593802834]
  We propose a generation framework inspired by narrative theory that decomposes narrative writing into subtasks tackled by specialized agents.<n>We show that Agents' Room generates stories preferred by expert evaluators over those produced by baseline systems.
  arXiv  Detail & Related papers  (2024-10-03T15:44:42Z)
• StoryVerse: Towards Co-authoring Dynamic Plot with LLM-based Character Simulation via Narrative Planning [8.851718319632973]
  Large Language Models (LLMs) drive the behavior of virtual characters, allowing plots to emerge from interactions between characters and their environments. We propose a novel plot creation workflow that mediates between a writer's authorial intent and the emergent behaviors from LLM-driven character simulation. The process creates "living stories" that dynamically adapt to various game world states, resulting in narratives co-created by the author, character simulation, and player.
  arXiv  Detail & Related papers  (2024-05-17T23:04:51Z)
• NarrativePlay: Interactive Narrative Understanding [27.440721435864194]
  We introduce NarrativePlay, a novel system that allows users to role-play a fictional character and interact with other characters in narratives in an immersive environment. We leverage Large Language Models (LLMs) to generate human-like responses, guided by personality traits extracted from narratives. NarrativePlay has been evaluated on two types of narratives, detective and adventure stories, where users can either explore the world or improve their favorability with the narrative characters through conversations.
  arXiv  Detail & Related papers  (2023-10-02T13:24:00Z)
• Telling Creative Stories Using Generative Visual Aids [52.623545341588304]
  We asked writers to write creative stories from a starting prompt, and provided them with visuals created by generative AI models from the same prompt. Compared to a control group, writers who used the visuals as story writing aid wrote significantly more creative, original, complete and visualizable stories. Findings indicate that cross modality inputs by AI can benefit divergent aspects of creativity in human-AI co-creation, but hinders convergent thinking.
  arXiv  Detail & Related papers  (2021-10-27T23:13:47Z)
• Wordcraft: a Human-AI Collaborative Editor for Story Writing [10.028560442375914]
  We propose Wordcraft, an AI-assisted editor for story writing in which a writer and a dialog system collaborate to write a story. Our novel interface uses few-shot learning and the natural affordances of conversation to support a variety of interactions.
  arXiv  Detail & Related papers  (2021-07-15T16:18:27Z)
• Deep Reinforcement Learning with Stacked Hierarchical Attention for Text-based Games [64.11746320061965]
  We study reinforcement learning for text-based games, which are interactive simulations in the context of natural language. We aim to conduct explicit reasoning with knowledge graphs for decision making, so that the actions of an agent are generated and supported by an interpretable inference procedure. We extensively evaluate our method on a number of man-made benchmark games, and the experimental results demonstrate that our method performs better than existing text-based agents.
  arXiv  Detail & Related papers  (2020-10-22T12:40:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.