InstructAV2AV: Instruction-Guided Audio-Video Joint Editing

• URL: http://arxiv.org/abs/2605.18467v1
• Date: Mon, 18 May 2026 14:27:05 GMT
• Title: InstructAV2AV: Instruction-Guided Audio-Video Joint Editing
• Abstract summary: We propose InstructAV2AV, the first end-to-end framework for instruction-guided audio-video joint editing.<n>We first develop a scalable data synthesis pipeline and construct InsAVE, the first large-scale audio-video editing dataset.<n>InstructAV2AV outperforms state-of-the-art methods across 11 metrics spanning three aspects on two evaluation sets.
• Score: 51.67847766136283
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent diffusion-based methods have achieved impressive progress in video content manipulation. However, they typically ignore the accompanying audio, leaving the audio disjointed from the edited results. In this paper, we propose InstructAV2AV, the first end-to-end framework for instruction-guided audio-video joint editing. We first develop a scalable data synthesis pipeline and construct InsAVE-80K, the first large-scale audio-video editing dataset with high-quality source-to-target pairs. With this data foundation, we adapt an audio-video generation backbone to leverage its robust priors. We concatenate the audio-video input with noisy latent codes to anchor the source context, propose the source-instruction gated attention to improve instruction following and content preservation, and introduce a two-stage training strategy to effectively transfer these pre-trained priors. Extensive experiments demonstrate that InstructAV2AV outperforms state-of-the-art methods across 11 metrics spanning three aspects on two evaluation sets, highlighting its potential for controllable content creation. Project page: https://hjzheng.net/projects/InstructAV2AV/.

Related papers

• AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation [47.70199516002865]
  We present textbfAVTok, a novel unified tokenizer designated for holistic audio-video generation.<n>AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries.<n>Experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation.
  arXiv  Detail & Related papers  (2026-06-29T18:35:17Z)
• Diffusion Models for Joint Audio-Video Generation [51.56484100374058]
  I release two high-quality, paired audio-video datasets.<n>I train the MM-Diffusion architecture from scratch on our datasets.<n>I propose a sequential two-step text-to-audio-video generation pipeline.
  arXiv  Detail & Related papers  (2026-03-17T03:31:37Z)
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning [44.518249924335045]
  Perception Audiovisual, PE-AV, is a new family of encoders for audio and video understanding trained with scaled contrastive learning.<n>Built on PE, PE-AV makes several key contributions to extend representations to audio, and supports joint embeddings across audio-video, audio-text, and video-text modalities.
  arXiv  Detail & Related papers  (2025-12-22T18:59:07Z)
• Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation [20.446421146630474]
  We introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising.<n>Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony.
  arXiv  Detail & Related papers  (2025-12-02T06:31:38Z)
• UniVerse-1: Unified Audio-Video Generation via Stitching of Experts [59.38012380516272]
  We introduce UniVerse-1, a unified, Veo-3-like model capable of simultaneously generating coordinated audio and video.<n>To enhance training efficiency, we bypass training from scratch and instead employ a stitching of experts (SoE) technique.
  arXiv  Detail & Related papers  (2025-09-07T17:55:03Z)
• SpA2V: Harnessing Spatial Auditory Cues for Audio-driven Spatially-aware Video Generation [50.03810359300705]
  SpA2V decomposes the generation process into two stages: audio-guided video planning and layout-grounded video generation.<n>We show that SpA2V excels in generating realistic videos with semantic and spatial alignment to the input audios.
  arXiv  Detail & Related papers  (2025-08-01T17:05:04Z)
• Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance [33.1393328136321]
  We propose a step-by-step video-to-audio (V2A) generation method for finer controllability over the generation process and more realistic audio synthesis.<n>Inspired by traditional Foley, our approach aims to capture all sound events induced by a video through the incremental generation of missing sound events.
  arXiv  Detail & Related papers  (2025-06-26T04:20:08Z)
• Quality Over Quantity? LLM-Based Curation for a Data-Efficient Audio-Video Foundation Model [16.22530358172138]
  The framework implements a scoring mechanism for selecting aligned training data segments.<n>It integrates Whisper, a speech-based foundation model, for audio and DINOv2 for video analysis in a dual-encoder structure.<n> Evaluations on AudioCaps, VALOR, and VGGSound demonstrate the effectiveness of the proposed model architecture.
  arXiv  Detail & Related papers  (2025-03-12T09:48:38Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)
• AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations [88.30635799280923]
  We introduce AV-data2vec which builds audio-visual representations based on predicting contextualized representations. Results on LRS3 show that AV-data2vec consistently outperforms existing methods with the same amount of data and model size.
  arXiv  Detail & Related papers  (2023-02-10T02:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.