FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts

• URL: http://arxiv.org/abs/2603.19857v1
• Date: Fri, 20 Mar 2026 11:19:29 GMT
• Title: FoleyDirector: Fine-Grained Temporal Steering for Video-to-Audio Generation via Structured Scripts
• Abstract summary: FoleyDirector is a framework that enables precise temporal guidance in DiT-based V2A generation.<n>It preserves the base model's audio quality and allows seamless switching between V2A generation and temporally controlled synthesis.
• Score: 33.4228845956243
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent Video-to-Audio (V2A) methods have achieved remarkable progress, enabling the synthesis of realistic, high-quality audio. However, they struggle with fine-grained temporal control in multi-event scenarios or when visual cues are insufficient, such as small regions, off-screen sounds, or occluded or partially visible objects. In this paper, we propose FoleyDirector, a framework that, for the first time, enables precise temporal guidance in DiT-based V2A generation while preserving the base model's audio quality and allowing seamless switching between V2A generation and temporally controlled synthesis. FoleyDirector introduces Structured Temporal Scripts (STS), a set of captions corresponding to short temporal segments, to provide richer temporal information. These features are integrated via the Script-Guided Temporal Fusion Module, which employs Temporal Script Attention to fuse STS features coherently. To handle complex multi-event scenarios, we further propose Bi-Frame Sound Synthesis, enabling parallel in-frame and out-of-frame audio generation and improving controllability. To support training and evaluation, we construct the DirectorSound dataset and introduce VGGSoundDirector and DirectorBench. Experiments demonstrate that FoleyDirector substantially enhances temporal controllability while maintaining high audio fidelity, empowering users to act as Foley directors and advancing V2A toward more expressive and controllable generation.

Related papers

• BackgroundMellow: A Multi-Modal Cohesive Framework for Narrative-Driven Rich Cinematic Soundscape Generation [1.8763872698583384]
  We present BackgroundMellow, a framework that treats story-to-audio generation as a precise orchestration and signal processing problem.<n>This framework is enabled without ground-truth through a master-specialist agent architecture that decomposes text into precise and multi-layered audio cues.<n>Our pipeline is built over Tango2 latent diffusion model for environmental synthesis and a novel Cinematic BGM Retriever mined from professional soundtracks.
  arXiv  Detail & Related papers  (2026-07-13T10:28:15Z)
• MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control [66.04301887685004]
  Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting.<n>We propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control.<n> MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking.
  arXiv  Detail & Related papers  (2026-06-28T16:01:04Z)
• FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision [22.5140828601393]
  FoleyGenEx is a unified video-to-audio (VTA) framework integrating multi-modal control, frame-level temporal alignment, and fine-grained semantics.<n>FoleyGenEx fills this gap via three core innovations: a conditional injection mechanism for audio-controlled VTA and Foley extension, a multi-modal dynamic masking strategy, and an adverb-based data augmentation algorithm.
  arXiv  Detail & Related papers  (2026-06-12T02:51:21Z)
• OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video [59.391671069156274]
  This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts.<n>We present OmniScript, an 8B- parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension.
  arXiv  Detail & Related papers  (2026-04-13T07:19:27Z)
• Foley Control: Aligning a Frozen Latent Text-to-Audio Model to Video [39.74394488889939]
  Foley Control is a lightweight approach to video-guided Foley.<n>It keeps pretrained single-modality models frozen and learns only a small cross-attention bridge between them.<n>Although we focus on Video-to-Foley, the same bridge design can potentially extend to other audio modalities.
  arXiv  Detail & Related papers  (2025-10-24T15:49:54Z)
• AudioStory: Generating Long-Form Narrative Audio with Large Language Models [87.23256929520743]
  AudioStory is a framework that integrates large language models with text-to-audio systems to generate structured, long-form audio narratives.<n>It employs LLMs to decompose complex narrative queries into temporally ordered sub-tasks with contextual cues.<n>Extensive experiments show the superiority of AudioStory on both single-audio generation and narrative audio generation, surpassing prior TTA baselines in both instruction-following ability and audio fidelity.
  arXiv  Detail & Related papers  (2025-08-27T17:55:38Z)
• Exploiting Temporal Audio-Visual Correlation Embedding for Audio-Driven One-Shot Talking Head Animation [62.218932509432314]
  Inherently, the temporal relationship of adjacent audio clips is highly correlated with that of the corresponding adjacent video frames.<n>We learn audio-visual correlations and integrate the correlations to help enhance feature representation and regularize final generation.
  arXiv  Detail & Related papers  (2025-04-08T07:23:28Z)
• ReelWave: Multi-Agentic Movie Sound Generation through Multimodal LLM Conversation [72.22243595269389]
  This paper proposes a multi-agentic framework for audio generation supervised by an autonomous Sound Director agent.<n>The Foley Artist works cooperatively with the Composer and Voice Actor agents, and together they autonomously generate off-screen sound to complement the overall production.<n>Our framework can generate rich and relevant audio content conditioned on video clips extracted from movies.
  arXiv  Detail & Related papers  (2025-03-10T11:57:55Z)
• FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment [11.796771978828403]
  We introduce FolAI, a two-stage generative framework that produces temporally coherent and semantically controllable sound effects from video.<n>Results show that our model reliably produces audio that is temporally aligned with visual motion, semantically consistent with user intent, and perceptually realistic.<n>These findings highlight the potential of FolAI as a controllable and modular solution for scalable, high-quality Foley sound synthesis in professional and interactive settings.
  arXiv  Detail & Related papers  (2024-12-19T16:37:19Z)
• Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Composition [72.22243595269389]
  We introduce Audio-Agent, a framework for audio generation, editing and composition based on text or video inputs.<n>In our method, we utilize a pre-trained TTA diffusion network as the audio generation agent to work in tandem with GPT-4.<n>For video-to-audio (VTA) tasks, most existing methods require training a timestamp detector to synchronize video events with the generated audio.
  arXiv  Detail & Related papers  (2024-10-04T11:40:53Z)
• Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound [19.694770666874827]
  Foley sound synthesis is crucial for multimedia production, enhancing user experience by synchronizing audio and video.<n>Recent studies on automating this labor-intensive process through video-to-sound generation face significant challenges.<n>We propose Video-Foley, a video-to-sound system using Root Mean Square (RMS) as an intuitive condition with semantic timbre prompts.
  arXiv  Detail & Related papers  (2024-08-21T18:06:15Z)
• FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds [14.636030346325578]
  We study Neural Foley, the automatic generation of high-quality sound effects synchronizing with videos, enabling an immersive audio-visual experience. We propose FoleyCrafter, a novel framework that leverages a pre-trained text-to-audio model to ensure high-quality audio generation. One notable advantage of FoleyCrafter is its compatibility with text prompts, enabling the use of text descriptions to achieve controllable and diverse video-to-audio generation according to user intents.
  arXiv  Detail & Related papers  (2024-07-01T17:35:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.