Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits

• URL: http://arxiv.org/abs/2512.07209v1
• Date: Mon, 08 Dec 2025 06:45:11 GMT
• Title: Coherent Audio-Visual Editing via Conditional Audio Generation Following Video Edits
• Authors: Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji,
• Abstract summary: We introduce a novel pipeline for joint audio-visual editing that enhances the coherence between edited video and its accompanying audio.<n>Our approach first applies state-of-the-art video editing techniques to produce the target video, then performs audio editing to align with the visual changes.
• Score: 33.1393328136321
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a novel pipeline for joint audio-visual editing that enhances the coherence between edited video and its accompanying audio. Our approach first applies state-of-the-art video editing techniques to produce the target video, then performs audio editing to align with the visual changes. To achieve this, we present a new video-to-audio generation model that conditions on the source audio, target video, and a text prompt. We extend the model architecture to incorporate conditional audio input and propose a data augmentation strategy that improves training efficiency. Furthermore, our model dynamically adjusts the influence of the source audio based on the complexity of the edits, preserving the original audio structure where possible. Experimental results demonstrate that our method outperforms existing approaches in maintaining audio-visual alignment and content integrity.

Related papers

• Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removal [90.14887235360611]
  SAVEBench is a paired audiovisual dataset with text and mask conditions to enable object-grounded source-to-target learning.<n>SAVE incorporates a Schrodinger Bridge that learns a direct transport from source to target audiovisual mixtures.<n>Our evaluation demonstrates that the proposed SAVE model is able to remove the target objects in audio and visual content while preserving the remaining content.
  arXiv  Detail & Related papers  (2025-12-14T23:19:15Z)
• Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner [66.96392168346851]
  AVI-Edit is a framework for audio-sync video instance editing.<n>We propose a granularity-aware mask refiner that iteratively refines coarse user-provided masks into precise instance-level regions.<n>We also design a self-feedback audio agent to curate high-quality audio guidance, providing fine-grained temporal control.
  arXiv  Detail & Related papers  (2025-12-11T11:58:53Z)
• AV-Edit: Multimodal Generative Sound Effect Editing via Audio-Visual Semantic Joint Control [10.55114688654566]
  AV-Edit is a generative sound effect editing framework that enables fine-grained editing of existing audio tracks in videos.<n>The proposed method employs a specially designed contrastive audio-visual masking autoencoder (CAV-MAE-Edit) for multimodal pre-training.<n> Experiments demonstrate that the proposed AV-Edit generates high-quality audio with precise modifications based on visual content.
  arXiv  Detail & Related papers  (2025-11-26T07:59:53Z)
• Object-AVEdit: An Object-level Audio-Visual Editing Model [79.62095842136115]
  We present textbfObject-AVEdit, achieving the object-level audio-visual editing based on the inversion-regeneration paradigm.<n>To achieve the object-level controllability during editing, we develop a word-to-sounding-object well-aligned audio generation model.<n>To achieve the better structural information preservation and object-level editing effect, we propose an inversion-regeneration holistically-optimized editing algorithm.
  arXiv  Detail & Related papers  (2025-09-27T18:12:13Z)
• ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editing [47.14083940177122]
  ThinkSound is a novel framework that enables stepwise, interactive audio generation and editing for videos.<n>Our approach decomposes the process into three complementary stages: semantically coherent, interactive object-centric refinement, and targeted editing.<n> Experiments demonstrate that ThinkSound achieves state-of-the-art performance in video-to-audio generation across both audio metrics and CoT metrics.
  arXiv  Detail & Related papers  (2025-06-26T16:32:06Z)
• Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation [56.92841782969847]
  We introduce a novel task called language-guided joint audio-visual editing. Given an audio and image pair of a sounding event, this task aims at generating new audio-visual content by editing the given sounding event conditioned on the language guidance. We propose a new diffusion-based framework for joint audio-visual editing and introduce two key ideas.
  arXiv  Detail & Related papers  (2024-10-09T22:02:30Z)
• AudioScenic: Audio-Driven Video Scene Editing [55.098754835213995]
  We introduce AudioScenic, an audio-driven framework designed for video scene editing. AudioScenic integrates audio semantics into the visual scene through a temporal-aware audio semantic injection process. We present an audio Magnitude Modulator module that adjusts the temporal dynamics of the scene in response to changes in audio magnitude. Second, the audio Frequency Fuser module is designed to ensure temporal consistency by aligning the frequency of the audio with the dynamics of the video scenes.
  arXiv  Detail & Related papers  (2024-04-25T12:55:58Z)
• Audio Editing with Non-Rigid Text Prompts [24.008609489049206]
  We show that the proposed editing pipeline is able to create audio edits that remain faithful to the input audio. We explore text prompts that perform addition, style transfer, and in-painting.
  arXiv  Detail & Related papers  (2023-10-19T16:09:44Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.