NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaboration

• URL: http://arxiv.org/abs/2504.18448v1
• Date: Fri, 25 Apr 2025 16:01:48 GMT
• Title: NoiseController: Towards Consistent Multi-view Video Generation via Noise Decomposition and Collaboration
• Abstract summary: We propose Multi-Level Noise DeController, Multi-Frame Noise DeController, and Joint Denoising to enhance consistencies in video generation.<n>We evaluate our NoiseController on public datasets on video generation and downstream tasks, demonstrating its state-of-the-art performance.
• Score: 25.13251765490759
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-quality video generation is crucial for many fields, including the film industry and autonomous driving. However, generating videos with spatiotemporal consistencies remains challenging. Current methods typically utilize attention mechanisms or modify noise to achieve consistent videos, neglecting global spatiotemporal information that could help ensure spatial and temporal consistency during video generation. In this paper, we propose the NoiseController, consisting of Multi-Level Noise Decomposition, Multi-Frame Noise Collaboration, and Joint Denoising, to enhance spatiotemporal consistencies in video generation. In multi-level noise decomposition, we first decompose initial noises into scene-level foreground/background noises, capturing distinct motion properties to model multi-view foreground/background variations. Furthermore, each scene-level noise is further decomposed into individual-level shared and residual components. The shared noise preserves consistency, while the residual component maintains diversity. In multi-frame noise collaboration, we introduce an inter-view spatiotemporal collaboration matrix and an intra-view impact collaboration matrix , which captures mutual cross-view effects and historical cross-frame impacts to enhance video quality. The joint denoising contains two parallel denoising U-Nets to remove each scene-level noise, mutually enhancing video generation. We evaluate our NoiseController on public datasets focusing on video generation and downstream tasks, demonstrating its state-of-the-art performance.

Related papers

• Track the Noise, Move the World:3D-Grounded Motion-Consistent Noise for Controllable Video Generation [14.852131707037655]
  We present UniCaMo, a unified framework that enables simultaneous control of object trajectories and camera viewpoints.<n>UniCaMo builds a shared 3D-grounded motion-consistent noise space across latent video frames.<n>It achieves state-of-the-art results in both video quality and motion controllability on standard controllable video generation benchmarks.
  arXiv  Detail & Related papers  (2026-07-02T22:17:20Z)
• CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation [57.47993858752813]
  We present CineOrchestra, a unified video diffusion model that controls subjects, events, cameras, and shot transitions simultaneously.<n>Our key insight is that these heterogeneous cinematic elements share a fundamental structure.<n>CineOrchestra outperforms six per-axis specialists on dense caption following and shot-transition timing.
  arXiv  Detail & Related papers  (2026-06-11T17:59:10Z)
• OmniCamera: A Unified Framework for Multi-task Video Generation with Arbitrary Camera Control [49.41924736941193]
  Video fundamentally intertwines two crucial axes: the dynamic content of a scene and the camera motion through which it is observed.<n>Existing generation models often entangle these factors, limiting independent control.<n>We introduce OmniCamera, a unified framework designed to explicitly disentangle and command these two dimensions.
  arXiv  Detail & Related papers  (2026-04-07T16:06:02Z)
• ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generation [55.76423101183408]
  ViSAudio is an end-to-end framework that employs conditional flow matching with a dual-branch audio generation architecture.<n>It generates high-quality audio with spatial immersion that adapts to viewpoint changes, sound-source motion, and diverse acoustic environments.
  arXiv  Detail & Related papers  (2025-12-02T18:56:12Z)
• Rolling Forcing: Autoregressive Long Video Diffusion in Real Time [86.40480237741609]
  Rolling Forcing is a novel video generation technique that enables streaming long videos with minimal error accumulation.<n>Rolling Forcing comes with three novel designs. First, instead of iteratively sampling individual frames, which accelerates error propagation, we design a joint denoising scheme.<n>Second, we introduce the attention sink mechanism into the long-horizon stream video generation task, which allows the model to keep key value states of initial frames as a global context anchor.<n>Third, we design an efficient training algorithm that enables few-step distillation over largely extended denoising windows.
  arXiv  Detail & Related papers  (2025-09-29T17:57:14Z)
• InfiniteTalk: Audio-driven Video Generation for Sparse-Frame Video Dubbing [66.48064661467781]
  We introduce sparse-frame video dubbing, a novel paradigm that strategically preserves references to maintain identity, iconic gestures, and camera trajectories.<n>We propose InfiniteTalk, a streaming audio-driven generator designed for infinite-length long sequence dubbing.<n> Comprehensive evaluations on HDTF, CelebV-HQ, and EMTD datasets demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-08-19T17:55:23Z)
• Audio-Assisted Face Video Restoration with Temporal and Identity Complementary Learning [56.62425904247682]
  We propose a General Audio-assisted face Video restoration Network (GAVN) to address various types of streaming video distortions.<n>GAVN first captures inter-frame temporal features in the low-resolution space to restore frames coarsely and save computational cost.<n>Finally, the reconstruction module integrates temporal features and identity features to generate high-quality face videos.
  arXiv  Detail & Related papers  (2025-08-06T07:38:27Z)
• SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers [25.36460340267922]
  We present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos.<n>Our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs.
  arXiv  Detail & Related papers  (2025-06-01T04:27:13Z)
• Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled Sampling [81.37449968164692]
  We propose Synchronized Coupled Sampling (SynCoS), a novel inference framework that synchronizes denoising paths across the entire video. Our approach combines two complementary sampling strategies, which ensure seamless local transitions and enforce global coherence. Extensive experiments show that SynCoS significantly improves multi-event long video generation, achieving smoother transitions and superior long-range coherence.
  arXiv  Detail & Related papers  (2025-03-11T16:43:45Z)
• Tuning-Free Long Video Generation via Global-Local Collaborative Diffusion [22.988212617368095]
  We propose GLC-Diffusion, a tuning-free method for long video generation.<n>It models the long video denoising process by establishing Global-Local Collaborative Denoising.<n>We also propose a Video Motion Consistency Refinement (VMCR) module that computes the gradient of pixel-wise and frequency-wise losses.
  arXiv  Detail & Related papers  (2025-01-08T05:49:39Z)
• Video Set Distillation: Information Diversification and Temporal Densification [68.85010825225528]
  Video textbfsets have two dimensions of redundancies: within-sample and inter-sample redundancies. We are the first to study Video Set Distillation, which synthesizes optimized video data by addressing within-sample and inter-sample redundancies.
  arXiv  Detail & Related papers  (2024-11-28T05:37:54Z)
• IDOL: Unified Dual-Modal Latent Diffusion for Human-Centric Joint Video-Depth Generation [136.5813547244979]
  We present IDOL (unIfied Dual-mOdal Latent diffusion) for high-quality human-centric joint video-depth generation. Our IDOL consists of two novel designs. First, to enable dual-modal generation and maximize the information exchange between video and depth generation. Second, to ensure a precise video-depth spatial alignment, we propose a motion consistency loss that enforces consistency between the video and depth feature motion fields.
  arXiv  Detail & Related papers  (2024-07-15T17:36:54Z)
• CoNo: Consistency Noise Injection for Tuning-free Long Video Diffusion [15.013908857230966]
  "Look-back" mechanism enhances the fine-grained scene transition between different video clips. Long-term consistency regularization focuses on explicitly minimizing the pixel-wise distance between the predicted noises of the extended video clip and the original one. Experiments have shown the effectiveness of the strategies by performing long-video generation under both single- and multi-text prompt conditions.
  arXiv  Detail & Related papers  (2024-06-07T16:56:42Z)
• TRIP: Temporal Residual Learning with Image Noise Prior for Image-to-Video Diffusion Models [94.24861019513462]
  TRIP is a new recipe of image-to-video diffusion paradigm. It pivots on image noise prior derived from static image to jointly trigger inter-frame relational reasoning. Extensive experiments on WebVid-10M, DTDB and MSR-VTT datasets demonstrate TRIP's effectiveness.
  arXiv  Detail & Related papers  (2024-03-25T17:59:40Z)
• Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners [69.70590867769408]
  Video and audio content creation serves as the core technique for the movie industry and professional users. Existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation.
  arXiv  Detail & Related papers  (2024-02-27T17:57:04Z)
• Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusion [30.631733395175765]
  Video multimodal fusion aims to integrate multimodal signals in videos. Video has longer multimodal sequences with more redundancy and noise in visual and audio modalities. We propose a denoising bottleneck fusion model for fine-grained video fusion.
  arXiv  Detail & Related papers  (2023-05-24T02:39:43Z)
• VideoFusion: Decomposed Diffusion Models for High-Quality Video Generation [88.49030739715701]
  This work presents a decomposed diffusion process via resolving the per-frame noise into a base noise that is shared among all frames and a residual noise that varies along the time axis. Experiments on various datasets confirm that our approach, termed as VideoFusion, surpasses both GAN-based and diffusion-based alternatives in high-quality video generation.
  arXiv  Detail & Related papers  (2023-03-15T02:16:39Z)
• Learning Task-Oriented Flows to Mutually Guide Feature Alignment in Synthesized and Real Video Denoising [137.5080784570804]
  Video denoising aims at removing noise from videos to recover clean ones. Some existing works show that optical flow can help the denoising by exploiting the additional spatial-temporal clues from nearby frames. We propose a new multi-scale refined optical flow-guided video denoising method, which is more robust to different noise levels.
  arXiv  Detail & Related papers  (2022-08-25T00:09:18Z)
• Sound2Sight: Generating Visual Dynamics from Sound and Context [36.38300120482868]
  We present Sound2Sight, a deep variational framework, that is trained to learn a per frame prior conditioned on a joint embedding of audio and past frames. To improve the quality and coherence of the generated frames, we propose a multimodal discriminator. Our experiments demonstrate that Sound2Sight significantly outperforms the state of the art in the generated video quality.
  arXiv  Detail & Related papers  (2020-07-23T16:57:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.