OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films

• URL: http://arxiv.org/abs/2608.04224v1
• Date: Tue, 04 Aug 2026 21:00:57 GMT
• Title: OmniVR: Joint Video-Audio Conditional Generation for Restoring Degraded Historical Films
• Abstract summary: Historical films suffer from co-occurring visual and audio degradations.<n>We present OmniVR, the first joint audio-video generative restoration model.
• Score: 77.65090601396003
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Historical films suffer from co-occurring visual and audio degradations---blur, noise, flicker, hiss, clipping, and dropout---yet existing methods restore each modality independently, leaving quality gaps and cross-modal inconsistency. We present OmniVR, the first joint audio-video generative restoration model. Built upon a 22B-parameter audio-video generation backbone, OmniVR formulates restoration as conditional generation within a unified multimodal DiT: the low-quality video and audio are encoded as latent conditions, combined with a fixed restoration prompt, and jointly denoised to recover visual structure, temporal motion, and acoustic detail under one coordinated objective. Three key designs enable this adaptation: (1) a joint audio-video degradation pipeline that simulates real old-film characteristics from Internet-collected data; (2) an architecture-preserving text-to-audio-video (T2AV) to audio-video-to-audio-video (AV2AV) transition with prompt annealing that maximally retains the generative prior; and (3) first-frame image-to-video (I2V) anchoring with loss reweighting and waveform supervision for long-video extrapolation and audio fidelity. We also propose OmniVRBench, the first benchmark that evaluates audio-video restoration across visual quality, audio quality, temporal consistency, and audio-visual synchrony on 200 real historical clips. OmniVR surpasses all prior methods on all six visual metrics, achieves the best audio quality, and produces natural colorization---the first method to jointly address all three aspects. Code and weights will be publicly released. Project Page: https://xin1u.github.io/OminiVR_PAGE/

Related papers

• AVTok: 1D Unified Tokenization for Holistic Audio-Video Generation [47.70199516002865]
  We present textbfAVTok, a novel unified tokenizer designated for holistic audio-video generation.<n>AVTok features a dual-stream transformer-based architecture with shared encoder-decoder and modal-specific learnable queries.<n>Experiments demonstrate that AVTok excels both in audio-video reconstruction and when integrated into downstream pipelines for audio-to-video, video-to-audio, and class-conditional joint audio-video generation.
  arXiv  Detail & Related papers  (2026-06-29T18:35:17Z)
• Diffusion Models for Joint Audio-Video Generation [51.56484100374058]
  I release two high-quality, paired audio-video datasets.<n>I train the MM-Diffusion architecture from scratch on our datasets.<n>I propose a sequential two-step text-to-audio-video generation pipeline.
  arXiv  Detail & Related papers  (2026-03-17T03:31:37Z)
• MOVA: Towards Scalable and Synchronized Video-Audio Generation [91.56945636522345]
  We introduce MOVA (MOSS Video and Audio), an open-source model capable of generating high-quality, synchronized audio-visual content.<n>By releasing the model weights and code, we aim to advance research and foster a vibrant community of creators.
  arXiv  Detail & Related papers  (2026-02-09T15:31:54Z)
• ALIVE: Animate Your World with Lifelike Audio-Video Generation [50.693986608051716]
  ALIVE is a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation.<n>To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch.<n>ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions.
  arXiv  Detail & Related papers  (2026-02-09T14:06:03Z)
• VABench: A Comprehensive Benchmark for Audio-Video Generation [22.00633729850902]
  VABench is a benchmark framework designed to evaluate the capabilities of synchronous audio-video generation.<n>It covers three task types: text-to-audio-video (T2AV), image-to-audio-video (I2AV), and stereo audio-video generation.<n>VABench covers seven major content categories: animals, human sounds, music, environmental sounds, synchronous physical sounds, complex scenes, and virtual worlds.
  arXiv  Detail & Related papers  (2025-12-10T03:57:29Z)
• ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation [55.76423101183408]
  ViSAudio is an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture.<n>It generates high-quality audio with spatial immersion that adapts to viewpoint changes, sound-source motion, and diverse acoustic environments.
  arXiv  Detail & Related papers  (2025-12-02T18:56:12Z)
• CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation [43.562848631392384]
  Audio-visual video segmentation aims to generate pixel-level maps of sound-producing objects within image frames. We propose a decoupled audio-video dependence combining audio and video features from their respective temporal and spatial dimensions.
  arXiv  Detail & Related papers  (2023-09-18T12:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.