AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation

• URL: http://arxiv.org/abs/2606.30811v1
• Date: Mon, 29 Jun 2026 18:35:17 GMT
• Title: AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation
• Abstract summary: We present textbfAVTok, a novel unified tokenizer designated for holistic audio-video generation.<n>AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries.<n>Experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation.
• Score: 47.70199516002865
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Audio-video generation has recently gained unprecedented research attention, aiming to synthesize high-quality sounding video content with fine-grained synchronization and semantic alignment between the auditory and visual components. The preceding methods predominantly adopt a dual-branch design with separate tokenization and generation modules per modality, neglecting the representation gap while necessitating intensive computational resources for proper training. Inspired by recent advancements in one-dimensional visual tokenization, we present \textbf{AVTok}, a novel unified tokenizer designated for holistic audio-video generation. AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries to efficiently and effectively encode an audio-video pair into a compact one-dimensional latent representation with a unified codebook. To cope with the heterogeneous information imbalance that hinders AVTok from exploiting aligned audio-visual information, we devise a hierarchical training strategy to progressively realize reconstruction capabilities for each modality. Extensive experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation. AVTok paves the way for the challenge of joint audio-video tokenization and provides a potential direction to build unified large multimodal models for audio-video generation.

Related papers

• OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation [73.57253966578186]
  We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations.<n>Experiments show that both objectives consistently improve the learnability of the latent spaces.
  arXiv  Detail & Related papers  (2026-07-26T21:51:28Z)
• InstructAV2AV: Instruction-Guided Audio-Video Joint Editing [51.67847766136283]
  We propose InstructAV2AV, the first end-to-end framework for instruction-guided audio-video joint editing.<n>We first develop a scalable data synthesis pipeline and construct InsAVE, the first large-scale audio-video editing dataset.<n>InstructAV2AV outperforms state-of-the-art methods across 11 metrics spanning three aspects on two evaluation sets.
  arXiv  Detail & Related papers  (2026-05-18T14:27:05Z)
• Foley-Flow: Coordinated Video-to-Audio Generation with Masked Audio-Visual Alignment and Dynamic Conditional Flows [75.44753202066171]
  Coordinated audio generation based on video inputs typically requires a strict audio-visual (AV) alignment.<n>We propose FoleyFlow to first align unimodal AV encoders via masked modeling training.<n>After training, the AV encoders which are separately pretrained using only unimodal data are aligned with semantic and rhythmic consistency.
  arXiv  Detail & Related papers  (2026-03-09T09:06:25Z)
• LTX-2: Efficient Joint Audio-Visual Foundation Model [3.1804093402153506]
  LTX-2 is an open-source model capable of generating temporally synchronized audiovisual content.<n>We employ a multilingual text encoder for broader prompt understanding.<n>LTX-2 produces rich, coherent audio tracks that follow the characters, environment, style, and emotion of each scene.
  arXiv  Detail & Related papers  (2026-01-06T18:24:41Z)
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning [44.518249924335045]
  Perception Audiovisual, PE-AV, is a new family of encoders for audio and video understanding trained with scaled contrastive learning.<n>Built on PE, PE-AV makes several key contributions to extend representations to audio, and supports joint embeddings across audio-video, audio-text, and video-text modalities.
  arXiv  Detail & Related papers  (2025-12-22T18:59:07Z)
• AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generation [24.799628787198397]
  AudioGen- Omni generates high-fidelity audio, speech, and song coherently synchronized with the input video.<n>Joint training paradigm integrates large-scale video-text-audio corpora.<n>Dense frame-level representations are fused using an AdaLN-based joint attention mechanism.<n>With an inference time of 1.91 seconds for 8 seconds of audio, it offers substantial improvements in both efficiency and generality.
  arXiv  Detail & Related papers  (2025-08-01T16:03:57Z)
• Step-by-Step Video-to-Audio Synthesis via Negative Audio Guidance [33.1393328136321]
  We propose a step-by-step video-to-audio (V2A) generation method for finer controllability over the generation process and more realistic audio synthesis.<n>Inspired by traditional Foley, our approach aims to capture all sound events induced by a video through the incremental generation of missing sound events.
  arXiv  Detail & Related papers  (2025-06-26T04:20:08Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• STELLA: Continual Audio-Video Pre-training with Spatio-Temporal Localized Alignment [61.83340833859382]
  Continuously learning a variety of audio-video semantics over time is crucial for audio-related reasoning tasks. This is a nontemporal problem and poses two critical challenges: sparse-temporal correlation between audio-video pairs and multimodal correlation overwriting that forgets audio-video relations. We propose a continual audio-video pre-training method with two novel ideas.
  arXiv  Detail & Related papers  (2023-10-12T10:50:21Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.