From Words to Worlds: Transforming One-line Prompt into Immersive Multi-modal Digital Stories with Communicative LLM Agent

• URL: http://arxiv.org/abs/2406.10478v2
• Date: Fri, 21 Jun 2024 08:09:17 GMT
• Title: From Words to Worlds: Transforming One-line Prompt into Immersive Multi-modal Digital Stories with Communicative LLM Agent
• Authors: Samuel S. Sohn, Danrui Li, Sen Zhang, Che-Jui Chang, Mubbasir Kapadia,
• Abstract summary: This paper introduces the StoryAgent framework to automate and refine digital storytelling. StoryAgent tackles key issues such as manual intervention, interactive scene orchestration, and narrative consistency. Results demonstrate the framework's capability to produce coherent digital stories without reference videos.
• Score: 11.553884271082127
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Digital storytelling, essential in entertainment, education, and marketing, faces challenges in production scalability and flexibility. The StoryAgent framework, introduced in this paper, utilizes Large Language Models and generative tools to automate and refine digital storytelling. Employing a top-down story drafting and bottom-up asset generation approach, StoryAgent tackles key issues such as manual intervention, interactive scene orchestration, and narrative consistency. This framework enables efficient production of interactive and consistent narratives across multiple modalities, democratizing content creation and enhancing engagement. Our results demonstrate the framework's capability to produce coherent digital stories without reference videos, marking a significant advancement in automated digital storytelling.

Related papers

• StoryAgent: Customized Storytelling Video Generation via Multi-Agent Collaboration [88.94832383850533]
  We propose a multi-agent framework designed for Customized Storytelling Video Generation (CSVG) StoryAgent decomposes CSVG into distinct subtasks assigned to specialized agents, mirroring the professional production process. Specifically, we introduce a customized Image-to-Video (I2V) method, LoRA-BE, to enhance intra-shot temporal consistency. Our contributions include the introduction of StoryAgent, a versatile framework for video generation tasks, and novel techniques for preserving protagonist consistency.
  arXiv  Detail & Related papers  (2024-11-07T18:00:33Z)
• Agents' Room: Narrative Generation through Multi-step Collaboration [54.98886593802834]
  We propose a generation framework inspired by narrative theory that decomposes narrative writing into subtasks tackled by specialized agents. We show that Agents' Room generates stories preferred by expert evaluators over those produced by baseline systems.
  arXiv  Detail & Related papers  (2024-10-03T15:44:42Z)
• Story3D-Agent: Exploring 3D Storytelling Visualization with Large Language Models [57.30913211264333]
  We present Story3D-Agent, a pioneering approach that transforms provided narratives into 3D-rendered visualizations. By integrating procedural modeling, our approach enables precise control over multi-character actions and motions, as well as diverse decorative elements. We have thoroughly evaluated our Story3D-Agent to validate its effectiveness, offering a basic framework to advance 3D story representation.
  arXiv  Detail & Related papers  (2024-08-21T17:43:15Z)
• DataNarrative: Automated Data-Driven Storytelling with Visualizations and Texts [27.218934418961197]
  We introduce a novel task for data story generation and a benchmark containing 1,449 stories from diverse sources. To address the challenges of crafting coherent data stories, we propose a multiagent framework employing two LLM agents. While our agentic framework generally outperforms non-agentic counterparts in both model-based and human evaluations, the results also reveal unique challenges in data story generation.
  arXiv  Detail & Related papers  (2024-08-09T21:31:33Z)
• SEED-Story: Multimodal Long Story Generation with Large Language Model [66.37077224696242]
  SEED-Story is a novel method that leverages a Multimodal Large Language Model (MLLM) to generate extended multimodal stories. We propose multimodal attention sink mechanism to enable the generation of stories with up to 25 sequences (only 10 for training) in a highly efficient autoregressive manner. We present a large-scale and high-resolution dataset named StoryStream for training our model and quantitatively evaluating the task of multimodal story generation in various aspects.
  arXiv  Detail & Related papers  (2024-07-11T17:21:03Z)
• Improving Visual Storytelling with Multimodal Large Language Models [1.325953054381901]
  This paper presents a novel approach leveraging large language models (LLMs) and large vision-language models (LVLMs) We introduce a new dataset comprising diverse visual stories, annotated with detailed captions and multimodal elements. Our method employs a combination of supervised and reinforcement learning to fine-tune the model, enhancing its narrative generation capabilities.
  arXiv  Detail & Related papers  (2024-07-02T18:13:55Z)
• TARN-VIST: Topic Aware Reinforcement Network for Visual Storytelling [14.15543866199545]
  As a cross-modal task, visual storytelling aims to generate a story for an ordered image sequence automatically. We propose a novel method, Topic Aware Reinforcement Network for VIsual StoryTelling (TARN-VIST) In particular, we pre-extracted the topic information of stories from both visual and linguistic perspectives.
  arXiv  Detail & Related papers  (2024-03-18T08:01:23Z)
• StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuation [76.44802273236081]
  We develop a model StoryDALL-E for story continuation, where the generated visual story is conditioned on a source image. We show that our retro-fitting approach outperforms GAN-based models for story continuation and facilitates copying of visual elements from the source image. Overall, our work demonstrates that pretrained text-to-image synthesis models can be adapted for complex and low-resource tasks like story continuation.
  arXiv  Detail & Related papers  (2022-09-13T17:47:39Z)
• Cue Me In: Content-Inducing Approaches to Interactive Story Generation [74.09575609958743]
  We focus on the task of interactive story generation, where the user provides the model mid-level sentence abstractions. We present two content-inducing approaches to effectively incorporate this additional information. Experimental results from both automatic and human evaluations show that these methods produce more topically coherent and personalized stories.
  arXiv  Detail & Related papers  (2020-10-20T00:36:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.