OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation

• URL: http://arxiv.org/abs/2605.12480v1
• Date: Tue, 12 May 2026 17:56:59 GMT
• Title: OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generation
• Abstract summary: Recent advances in joint audio-video generation demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization.<n>These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results.<n>We propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations.
• Score: 60.95158352332569
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in joint audio-video generation have been remarkable, yet real-world applications demand strong per-modality fidelity, cross-modal alignment, and fine-grained synchronization. Reinforcement Learning (RL) offers a promising paradigm, but its extension to multi-objective and multi-modal joint audio-video generation remains unexplored. Notably, our in-depth analysis first reveals that the primary obstacles to applying RL in this stem from: (i) multi-objective advantages inconsistency, where the advantages of multimodal outputs are not always consistent within a group; (ii) multi-modal gradients imbalance, where video-branch gradients leak into shallow audio layers responsible for intra-modal generation; (iii) uniform credit assignment, where fine-grained cross-modal alignment regions fail to get efficient exploration. These shortcomings suggest that vanilla RL fine-tuning strategy with a single global advantage often leads to suboptimal results. To address these challenges, we propose OmniNFT, a novel modality-aware online diffusion RL framework with three key innovations: (1) Modality-wise advantage routing, which routes independent per-reward advantages to their respective modality generation branches. (2) Layer-wise gradient surgery, which selectively detaches video-branch gradients on shallow audio layers while retaining those for cross-modal interaction layers. (3) Region-wise loss reweighting, which modulates policy optimization toward critical regions related to audio-video synchronization and fine-grained alignment. Extensive experiments on JavisBench and VBench with the LTX-2 backbone demonstrate that OmniNFT achieves comprehensive improvements in audio and video perceptual quality, cross-modal alignment, and audio-video synchronization.

Related papers

• OmniVAE: An Audio-Video VAE with Cross-Modal Alignment for Joint Generation [73.57253966578186]
  We present OmniVAE, a jointly trained audio-video VAE that learns fine-grained semantic alignment between audio and video latent representations.<n>Experiments show that both objectives consistently improve the learnability of the latent spaces.
  arXiv  Detail & Related papers  (2026-07-26T21:51:28Z)
• Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling [59.97930201616015]
  We propose Talker-T2AV, an autoregressive diffusion framework where high-level cross-modal modeling occurs in a shared backbone.<n>A shared autoregressive language model jointly reasons over audio and video in a unified patch-level token space.<n> Experiments on talking portrait benchmarks show Talker-T2AV outperforms dual-branch baselines in lip-sync accuracy, video quality, and audio quality.
  arXiv  Detail & Related papers  (2026-04-26T07:48:47Z)
• OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering [53.30273345137238]
  We propose OmniJigsaw, a generic self-supervised framework built upon a temporal reordering proxy task.<n>Joint Modality Integration, Sample-level Modality Selection, and Clip-level Modality Masking are used to compel cross-modal integration.<n>Our analysis reveals a bi-modal shortcut phenomenon'' in joint modality integration and demonstrates that fine-grained clip-level modality masking mitigates this issue.
  arXiv  Detail & Related papers  (2026-04-09T13:09:40Z)
• Apollo: Unified Multi-Task Audio-Video Joint Generation [15.004783109205666]
  Non-commercial approaches still suffer audio-visual asynchrony, poor lip-speech alignment, and unimodal degradation.<n>We introduce Apollo and delve into three axes--model architecture, training strategy, and data curation.<n>For datasets, we present the first large-scale audio-video dataset with dense captions.
  arXiv  Detail & Related papers  (2026-01-07T18:03:45Z)
• UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactions [34.27531187147479]
  UniAVGen is a unified framework for joint audio and video generation.<n>UniAVGen delivers overall advantages in audiovideo synchronization, timbre, and emotion consistency.
  arXiv  Detail & Related papers  (2025-11-05T10:06:51Z)
• Complementary and Contrastive Learning for Audio-Visual Segmentation [74.11434759171199]
  We present Complementary and Contrastive Transformer (CCFormer), a novel framework adept at processing both local and global information.<n>Our method sets new state-of-the-art benchmarks across the S4, MS3 and AVSS datasets.
  arXiv  Detail & Related papers  (2025-10-11T06:36:59Z)
• Audio-Guided Dynamic Modality Fusion with Stereo-Aware Attention for Audio-Visual Navigation [41.85539404067887]
  In audio-visual navigation (AVN) tasks, an embodied agent must autonomously localize a sound source in complex 3D environments.<n>Existing methods often rely on static modality fusion strategies and neglect the spatial cues embedded in stereo audio.<n>We propose an end-to-end reinforcement learning-based AVN framework with two key innovations.
  arXiv  Detail & Related papers  (2025-09-21T05:11:09Z)
• Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models [35.86252379746625]
  The goal of this work is to enhance balanced multimodal understanding in audio-visual large language models (AV-LLMs)<n>In current AV-LLMs, audio and video features are typically processed jointly in the decoder.<n>We propose Fork-Merge Decoding (FMD), a simple yet effective inference-time strategy that requires no additional training or architectural modifications.
  arXiv  Detail & Related papers  (2025-05-27T08:22:56Z)
• Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration [50.38965090742822]
  Long-grained video-audio reasoning and fine-grained pixel impose conflicting requirements on omnimodal models.<n>We tackle this trade-off with a two-system architecture: a Global Reasoning System selects informatives and rewrites the task at low spatial cost, while a Detail Understanding System performs pixel-level grounding.<n>Because optimalhorizon'' selection and reformulation are ambiguous and hard to supervise, we formulate them as a reinforcement learning (RL) problem and present Omni-R1, an end-to-end RL framework built on Group Relative Policy.
  arXiv  Detail & Related papers  (2025-05-26T17:34:06Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.