SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning

• URL: http://arxiv.org/abs/2605.12179v1
• Date: Tue, 12 May 2026 14:22:13 GMT
• Title: SyncDPO: Enhancing Temporal Synchronization in Video-Audio Joint Generation via Preference Learning
• Abstract summary: We propose a post-training framework SyncDPO to improve the temporal sensitivity of V-A joint generation.<n>We demonstrate that SyncDPO significantly outperforms other methods in improving model's temporal alignment capability.
• Score: 17.14027883239169
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in video-audio joint generation have achieved remarkable success in semantic correspondence. However, achieving precise temporal synchronization, which requires fine-grained alignment between audio events and their visual triggers, remains a challenging problem. The post-training method for joint generation is largely dominated by Supervised Fine-Tuning, but the commonly used Mean Squared Error loss provides insufficient penalties for subtle temporal misalignments. Direct Preference Optimization offers an alternative by introducing explicit misaligned counterparts to better improve temporal sensitivity. In this paper we propose a post-training framework SyncDPO, leveraging DPO to improve the temporal sensitivity of V-A joint generation. Conventional DPO pipelines typically depend on costly sampling-and-ranking procedures to construct preference pairs, resulting in substantial computational cost. To improve efficiency, we introduce a suite of on-the-fly rule-based negative construction strategies that distort temporal structures without incurring additional annotation or sampling. We demonstrate that the temporal alignment capability can be effectively reinforced by providing explicit negative supervision through temporally distorted V-A pairs. Accordingly, we implement a curriculum learning strategy that progressively increases the difficulty of negative samples, transitioning from coarse misalignment to subtle inconsistencies. Extensive objective and subjective experiments across four diverse benchmarks, ranging from ambient sound videos to human speech videos, demonstrate that SyncDPO significantly outperforms other methods in improving model's temporal alignment capability. It also demonstrates superior generalization on out-of-distribution benchmark by capturing intrinsic motion-sound dynamics. Demo and code is available in https://syncdpo.github.io/syncdpo/.

Related papers

• Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion [33.237232061796796]
  temporally sparse artifacts such as motion collapse, object flickering, and color oversaturation remain a major barrier to perceptual realism.<n>Existing methods struggle with these issues due to two key limitations.<n>We propose concentrated Implicit Preference Optimization (cIPO), a post-training framework for video diffusion models.
  arXiv  Detail & Related papers  (2026-07-30T11:36:48Z)
• Probability-Conserving Flow Guidance [49.03107678763765]
  Adaptive Manifold Guidance (AdaMaG) improves realism, reduces hallucinations, and controlled desaturation in high-guidance.<n>Across image generation benchmarks, AdaMaG improves realism, reduces hallucinations, and controlled desaturation in high-guidance.
  arXiv  Detail & Related papers  (2026-05-19T16:34:01Z)
• Offline Preference Optimization for Rectified Flow with Noise-Tracked Pairs [17.05976471653904]
  We propose Prior Noise-Aware Preference Optimization (PNAPO), an off-policy alignment framework specialized for rectified flow.<n>PNAPO augments preference data by retaining the paired prior noises used to generate each winner/loser image.<n>Experiments on state-of-the-art RF T2I backbones show that PNAPO consistently improves preference metrics while substantially reducing training compute.
  arXiv  Detail & Related papers  (2026-05-10T09:13:40Z)
• TARO: Temporal Adversarial Rectification Optimization Using Diffusion Models as Purifiers [0.6875312133832079]
  Adversarial purification with diffusion models seeks to project adversarial examples back toward the data manifold.<n>Recent work shows that standard diffusion purification can fail under adaptive evaluation, while test-time score-based optimization is more resilient.<n>We propose Temporal Adversarial Rectification Optimization (TARO), an inference-time purification method that builds a temporally guided score prior to multiple denoising views.
  arXiv  Detail & Related papers  (2026-05-08T20:02:57Z)
• Diffusion-APO: Trajectory-Aware Direct Preference Alignment for Video Diffusion Transformers [12.948398661304184]
  Diffusion-APO is a trajectory-aware algorithm for aligning video diffusion models with human intent.<n>We introduce a unified and modular RLHF framework that integrates online ranking, half-online anchoring, offline refinement, and distillation-aware drift correction.<n>We demonstrate that Diffusion-APO consistently outperforms standard baselines in visual quality and instruction following.
  arXiv  Detail & Related papers  (2026-05-08T09:37:46Z)
• Bringing Order to Asynchronous SGD: Towards Optimality under Data-Dependent Delays with Momentum [16.032739611868685]
  Asynchronous distributed gradient training (SGD) enables scalable distributed training but suffers from staleness.<n>Existing mitigation strategies, such as delay-dependent learning rates and staleness-aware training, typically or discard delayed gradients.<n>We propose a momentum-based asynchronous framework designed to preserve information from delayed gradients while mitigating staleness.
  arXiv  Detail & Related papers  (2026-05-03T20:24:24Z)
• CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing [76.74048814837336]
  Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video.<n>Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level.<n>We propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT)
  arXiv  Detail & Related papers  (2026-04-14T05:03:57Z)
• TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation [0.0]
  This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework.<n>It explores few-step inference for efficient audio-driven talking-head generation.<n>The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker.
  arXiv  Detail & Related papers  (2026-03-06T09:09:01Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• READ: Real-time and Efficient Asynchronous Diffusion for Audio-driven Talking Head Generation [55.58089937219475]
  We propose READ, the first real-time diffusion-transformer-based talking head generation framework.<n>Our approach first learns highly compressed video latent space via a VAE, significantly reducing the token count to speech generation.<n>We show that READ outperforms state-of-the-art methods by generating competitive talking head videos with significantly reduced runtime.
  arXiv  Detail & Related papers  (2025-08-05T13:57:03Z)
• Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled Sampling [81.37449968164692]
  We propose Synchronized Coupled Sampling (SynCoS), a novel inference framework that synchronizes denoising paths across the entire video.<n>Our approach combines two complementary sampling strategies, which ensure seamless local transitions and enforce global coherence.<n>Extensive experiments show that SynCoS significantly improves multi-event long video generation, achieving smoother transitions and superior long-range coherence.
  arXiv  Detail & Related papers  (2025-03-11T16:43:45Z)
• Intrinsic Temporal Regularization for High-resolution Human Video Synthesis [59.54483950973432]
  temporal consistency is crucial for extending image processing pipelines to the video domain. We propose an effective intrinsic temporal regularization scheme, where an intrinsic confidence map is estimated via the frame generator to regulate motion estimation. We apply our intrinsic temporal regulation to single-image generator, leading to a powerful " INTERnet" capable of generating $512times512$ resolution human action videos.
  arXiv  Detail & Related papers  (2020-12-11T05:29:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.