UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors

• URL: http://arxiv.org/abs/2605.00658v1
• Date: Fri, 01 May 2026 13:40:56 GMT
• Title: UniVidX: A Unified Multimodal Framework for Versatile Video Generation via Diffusion Priors
• Abstract summary: We present UniVidX, a unified framework that leverages VDM priors for versatile video generation.<n>UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space.<n>It adapts to modality-specific distributions while preserving the backbone's native priors.
• Score: 26.743983108253943
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent progress has shown that video diffusion models (VDMs) can be repurposed for diverse multimodal graphics tasks. However, existing methods often train separate models for each problem setting, which fixes the input-output mapping and limits the modeling of correlations across modalities. We present UniVidX, a unified multimodal framework that leverages VDM priors for versatile video generation. UniVidX formulates pixel-aligned tasks as conditional generation in a shared multimodal space, adapts to modality-specific distributions while preserving the backbone's native priors, and promotes cross-modal consistency during synthesis. It is built on three key designs. Stochastic Condition Masking (SCM) randomly partitions modalities into clean conditions and noisy targets during training, enabling omni-directional conditional generation instead of fixed mappings. Decoupled Gated LoRA (DGL) introduces per-modality LoRAs that are activated when a modality serves as the generation target, preserving the strong priors of the VDM. Cross-Modal Self-Attention (CMSA) shares keys and values across modalities while keeping modality-specific queries, facilitating information exchange and inter-modal alignment. We instantiate UniVidX in two domains: UniVid-Intrinsic, for RGB videos and intrinsic maps including albedo, irradiance, and normal; and UniVid-Alpha, for blended RGB videos and their constituent RGBA layers. Experiments show that both models achieve performance competitive with state-of-the-art methods across distinct tasks and generalize robustly to in-the-wild scenarios, even when trained on fewer than 1,000 videos. Project page: https://houyuanchen111.github.io/UniVidX.github.io/

Related papers

• Paths: Prompt-aware Spatio-temporal Transformer with Hierarchical Multi-modal Fusion for RGB-Event Video Person Re-Identification [75.15027201505912]
  We propose a unified framework with disc-temporal modeling and hierarchical multi-modal fusion for RE-VReID.<n>Our framework can learn robust and discriminative representations for RE-VReID.
  arXiv  Detail & Related papers  (2026-08-13T10:58:47Z)
• Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos [70.78278988781882]
  We present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs.<n>Our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating.
  arXiv  Detail & Related papers  (2026-05-25T18:02:20Z)
• VINO: A Unified Visual Generator with Interleaved OmniModal Context [36.71641694179164]
  VINO is a unified visual generator that performs image and video generation and editing within a single framework.<n>Instead of relying on task-specific models or independent modules for each modality, VINO uses a shared diffusion backbone.
  arXiv  Detail & Related papers  (2026-01-05T18:56:34Z)
• X-ReID: Multi-granularity Information Interaction for Video-Based Visible-Infrared Person Re-Identification [79.37768038337971]
  We propose a novel cross-modality feature learning framework named X-ReID for VVI-ReID.<n> Specifically, we first propose a Cross-modality Prototype Collaboration (CPC)<n>Then, a Multi-granularity Information Interaction (MII) is designed, incorporating short-term interactions from adjacent frames, long-term cross-frame information fusion, and cross-modality feature alignment.
  arXiv  Detail & Related papers  (2025-11-22T07:57:15Z)
• UniVid: Unifying Vision Tasks with Pre-trained Video Generation Models [12.21686773633269]
  Large language models, trained on extensive corpora, successfully unify diverse linguistic tasks within a single generative framework.<n>Such modeling requires task-specific pre-training across modalities and sources, which is costly and limits scalability to unseen tasks.<n>We propose UniVid, a framework that fine-tunes a video diffusion transformer to handle various vision tasks without task-specific modifications.
  arXiv  Detail & Related papers  (2025-09-26T01:43:40Z)
• Unified Multimodal Discrete Diffusion [78.48930545306654]
  Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches.<n>We explore discrete diffusion models as a unified generative formulation in the joint text and image domain.<n>We present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images.
  arXiv  Detail & Related papers  (2025-03-26T17:59:51Z)
• Multi-Pair Temporal Sentence Grounding via Multi-Thread Knowledge Transfer Network [57.72095897427665]
  temporal sentence grounding (TSG) aims to locate query-relevant segments in videos.<n>Previous methods follow a single-thread framework that cannot co-train different pairs.<n>We propose Multi-Pair TSG, which aims to co-train these pairs.
  arXiv  Detail & Related papers  (2024-12-20T08:50:11Z)
• FM-ViT: Flexible Modal Vision Transformers for Face Anti-Spoofing [88.6654909354382]
  We present a pure transformer-based framework, dubbed the Flexible Modal Vision Transformer (FM-ViT) for face anti-spoofing. FM-ViT can flexibly target any single-modal (i.e., RGB) attack scenarios with the help of available multi-modal data. Experiments demonstrate that the single model trained based on FM-ViT can not only flexibly evaluate different modal samples, but also outperforms existing single-modal frameworks by a large margin.
  arXiv  Detail & Related papers  (2023-05-05T04:28:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.