Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds

• URL: http://arxiv.org/abs/2608.23383v2
• Date: Tue, 25 Aug 2026 14:42:59 GMT
• Title: Long-Horizon Audio-Visual Generation for Persistent Stories and Interactive Worlds
• Abstract summary: We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants.<n>The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio.<n>The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway.
• Score: 60.18437892555415
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video generation is progressing beyond isolated clips toward long-form narratives and interactive worlds, requiring models to preserve identities, follow user controls, and remain stable over extended rollouts. We present JoyAI-Echo-1.5, a unified audio-visual generation system with two purpose-built variants. The long-video variant introduces composable cross-shot memory that aggregates visual evidence across multiple prior shots and speaker cues derived from speech-filtered full-shot audio, enabling persistent character appearance and voice identity across flexible combinations of text, image, and memory conditioning. The world-model variant converts heterogeneous navigation inputs into calibrated metric 6-DoF camera trajectories and injects them through a geometry-aware conditioning pathway, enabling controller-agnostic interaction across flexible viewpoints. To support efficient long-horizon generation, we transform a bidirectional audio-visual backbone into a causal few-step generator using progressive teacher forcing and short- and long-horizon Self-Gradient Forcing on self-generated rollouts. Experiments demonstrate strong performance in both settings. JoyAI-Echo-1.5 achieves improvements over existing long-video baselines in cross-shot consistency, visual quality, text alignment, and speech fidelity. Its world-model variant ranks first on WBench, with an average score of 81.7, and achieves leading visual quality and long-horizon persistence on SANA-WM-Bench. Together, these results indicate that memory, geometric control, and rollout-aware training provide a practical foundation for generating coherent stories and continuously evolving interactive worlds. Project page: https://echo-team-joy-future-academy-jd.github.io/Echo-1.5-Page/.

Related papers

• DiVA: Enabling Interactive Digital Life Simulation via Video Models [107.640138694388]
  DiVA is an interactive digital life simulator pioneering a new paradigm for long-term, open-ended interactive experiences.<n>It maintains long-term visual quality and realism, validating its effectiveness as a sustainable, interactive simulation.
  arXiv  Detail & Related papers  (2026-09-12T09:33:15Z)
• EchoWM: Open and Enterable Omnimodal World Models [71.49148067556887]
  EchoWM is an omnimodal world model for enterable generative media.<n>It responds to continuous navigation while jointly generating 720p video, environmental sound, music and speech.
  arXiv  Detail & Related papers  (2026-08-24T12:39:59Z)
• Wonder: Video World Model Done Better [70.28870858365214]
  We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration.<n>Given an image or a conditional video, Wonder constructs a playable world where users can navigate interactively by moving the camera.<n>We introduce a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues.
  arXiv  Detail & Related papers  (2026-07-28T17:45:25Z)
• OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation [73.57253966578186]
  We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations.<n>Experiments show that both objectives consistently improve the learnability of the latent spaces.
  arXiv  Detail & Related papers  (2026-07-26T21:51:28Z)
• OmniMate: Open-Ended Real-Time Streaming Audio-Visual Generation for Interactive Avatars [13.53343552751081]
  We propose a unified framework for open-ended real-time interactive audio-visual avatar generation.<n> OmniMate jointly synthesizes visual content, speech, and sound effects in real time, enabling natural and immersive multi-turn interactions.<n>We show that OmniMate achieves high-quality, low-latency streaming generation while maintaining strong long-term audio-visual consistency.
  arXiv  Detail & Related papers  (2026-07-25T03:42:52Z)
• VividVoice: A Unified Framework for Scene-Aware Visually-Driven Speech Synthesis [33.07214721477614]
  We introduce and define a novel task-Scene-Aware Visually-Driven Speech Synthesis, aimed at addressing the limitations of existing speech generation models.<n>We propose VividVoice, a unified generative framework to tackle the two core challenges of data scarcity and modality decoupling.<n>We show that VividVoice significantly outperforms existing baseline models in terms of audio fidelity, content clarity, and multimodal consistency.
  arXiv  Detail & Related papers  (2026-02-01T07:56:24Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• LongScape: Advancing Long-Horizon Embodied World Models with Context-Aware MoE [16.561410415129778]
  LongScape is a hybrid framework that combines intra-chunk diffusion denoising with inter-chunk autoregressive causal generation.<n>Our core innovation is an action-guided, variable-length chunking mechanism that partitions video based on the semantic context of robotic actions.
  arXiv  Detail & Related papers  (2025-09-26T02:47:05Z)
• FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio [45.71036380866305]
  We abstract the process of people hearing speech, extracting meaningful cues, and creating dynamically audio-consistent talking faces from a single audio. Specifically, it involves two critical challenges: one is to effectively decouple identity, content, and emotion from entangled audio, and the other is to maintain intra-video diversity and inter-video consistency. We introduce the Controllable Coherent Frame generation, which involves the flexible integration of three trainable adapters with frozen Latent Diffusion Models.
  arXiv  Detail & Related papers  (2024-03-04T09:59:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.