UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating

• URL: http://arxiv.org/abs/2606.21661v1
• Date: Fri, 19 Jun 2026 18:06:15 GMT
• Title: UnityShots: Memory-Driven Multi-Shot Audio-Video Generation with Boundary-Aware Gating
• Abstract summary: We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3.<n>The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail.<n>The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank.
• Score: 60.20680928161925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating a coherent multi-shot video requires structured cross-shot memory. Subject appearance, scene context, and speaker identity must persist across cuts. Existing approaches either train end-to-end over fixed-length sequences and cannot scale, generate shot-by-shot with memory banks that grow linearly, or orchestrate pretrained generators under an LLM planner without a multi-shot-aware backbone. We present UnityShots, a memory-driven multi-shot audio-video generation system built on LTX-2.3, trained on annotated cinematic and music-video shots. The video stream maintains two fixed-size slots, a long-term memory (LTM) slot anchored to the opening shot and a short-term memory (STM) slot holding the immediately preceding tail, both updated at every cut by a boundary-conditioned gate that fuses visual cut probability and beat-tracker signals. The audio stream injects a reference speaker token at every shot to preserve vocal timbre without a sliding audio bank. A discrete cut-type prior, learned through AdaLN, becomes an inference-time control knob over transition strength. We release a benchmark of $200$ multi-cultural multi-shot sequences spanning six ethnic regions and ten or more languages, with per-shot reference identities, reference audio, and per-boundary transition labels. Evaluated across I2V, T2V, and R2V conditioning modes, UnityShots leads open-source baselines on every cross-shot coherence metric and matches the strongest closed-source system on the multi-shot axes.

Related papers

• UniSwap: Streaming Audio-Visual Identity Swapping for Talking Videos [23.032949794607106]
  We present UniSwap, the first framework for streaming joint audio-visual identity replacement in talking videos.<n>Experiments demonstrate strong audio-visual synchronization, competitive identity preservation, efficient streaming, and stable long-form generation.
  arXiv  Detail & Related papers  (2026-08-12T07:45:39Z)
• MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control [66.04301887685004]
  Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting.<n>We propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control.<n> MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking.
  arXiv  Detail & Related papers  (2026-06-28T16:01:04Z)
• TRADE: Transducer-Augmented Decoder for Speech LLM [52.69726809996728]
  Speech Large Language Models (Speech LLMs) lack a principled mechanism for streaming inference.<n>We propose TRADE TRansducer-Augmented DEcoder, which augments a multimodal LLM with a transducer branch that shares the audio encoder.<n>Three design choices make the system accurate, streamable, and long-form capable.
  arXiv  Detail & Related papers  (2026-06-07T07:15:34Z)
• ReCA: Multi-Shot Long Video Extrapolation via Recursive Context Allocation [69.45329019089041]
  Multi-Shot Video Extrapolation (MSVE) is a task that extends an observed frame or clip into a sequence of cinematically structured shots.<n>MSVE reveals that long-video failure is not merely a limitation of context length, but a failure of context allocation.<n>We propose Recursive Context Allocation (ReCA), an inference-time framework that allocates context hierarchically across planning and generation.
  arXiv  Detail & Related papers  (2026-05-26T04:16:56Z)
• LTX-2: Efficient Joint Audio-Visual Foundation Model [3.1804093402153506]
  LTX-2 is an open-source model capable of generating temporally synchronized audiovisual content.<n>We employ a multilingual text encoder for broader prompt understanding.<n>LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene.
  arXiv  Detail & Related papers  (2026-01-06T18:24:41Z)
• MultiShotMaster: A Controllable Multi-Shot Video Generation Framework [67.38203939500157]
  Current generation techniques excel at single-shot clips but struggle to produce narrative multi-shot videos.<n>We propose MultiShotMaster, a framework for highly controllable multi-shot video generation.
  arXiv  Detail & Related papers  (2025-12-02T18:59:48Z)
• SMART: Shot-Aware Multimodal Video Moment Retrieval with Audio-Enhanced MLLM [34.75500054479833]
  Video Moment Retrieval aims to localize a specific temporal segment in an untrimmed video based on a natural language query.<n>Most existing methods still rely on coarse temporal understanding and a single visual modality, limiting performance on complex videos.<n>We introduce textitShot-aware textitMulti textitAudio-enhanced textitRetrieval of textitSegments.
  arXiv  Detail & Related papers  (2025-11-18T05:03:17Z)
• Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video [39.74394488889939]
  Foley Control is a lightweight approach to video-guided Foley.<n>It keeps pretrained single-modality models frozen and learns only a small cross-attention bridge between them.<n>Although we focus on Video-to-Foley, the same bridge design can potentially extend to other audio modalities.
  arXiv  Detail & Related papers  (2025-10-24T15:49:54Z)
• When and How to Cut Classical Concerts? A Multimodal Automated Video Editing Approach [9.554646174100123]
  We address the challenge of editing multicamera recordings of classical music concerts by decomposing the problem into two key sub-tasks: when to cut and how to cut.<n>Building on recent literature, we propose a novel multimodal architecture for the temporal segmentation task (when to cut)<n>For the spatial selection task (how to cut), we improve the literature by updating from old backbones, e.g. ResNet, with a CLIP-based encoder and constraining distractor selection to segments from the same concert.
  arXiv  Detail & Related papers  (2025-10-07T08:18:27Z)
• VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention [76.3175166538482]
  VideoGen-of-Thought (VGoT) is a step-by-step framework that automates multi-shot video synthesis from a single sentence.<n>VGoT addresses three core challenges: Narrative fragmentation, visual inconsistency, and transition artifacts.<n>Combined in a training-free pipeline, VGoT surpasses strong baselines by 20.4% in within-shot face consistency and 17.4% in style consistency.
  arXiv  Detail & Related papers  (2024-12-03T08:33:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.