Learning Temporally Consistent Video Depth from Video Diffusion Priors

• URL: http://arxiv.org/abs/2406.01493v2
• Date: Tue, 4 Jun 2024 03:33:52 GMT
• Title: Learning Temporally Consistent Video Depth from Video Diffusion Priors
• Authors: Jiahao Shao, Yuanbo Yang, Hongyu Zhou, Youmin Zhang, Yujun Shen, Matteo Poggi, Yiyi Liao,
• Abstract summary: This work addresses the challenge of video depth estimation. We reformulate the prediction task into a conditional generation problem. This allows us to leverage the prior knowledge embedded in existing video generation models.
• Score: 57.929828486615605
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work addresses the challenge of video depth estimation, which expects not only per-frame accuracy but, more importantly, cross-frame consistency. Instead of directly developing a depth estimator from scratch, we reformulate the prediction task into a conditional generation problem. This allows us to leverage the prior knowledge embedded in existing video generation models, thereby reducing learning difficulty and enhancing generalizability. Concretely, we study how to tame the public Stable Video Diffusion (SVD) to predict reliable depth from input videos using a mixture of image depth and video depth datasets. We empirically confirm that a procedural training strategy -- first optimizing the spatial layers of SVD and then optimizing the temporal layers while keeping the spatial layers frozen -- yields the best results in terms of both spatial accuracy and temporal consistency. We further examine the sliding window strategy for inference on arbitrarily long videos. Our observations indicate a trade-off between efficiency and performance, with a one-frame overlap already producing favorable results. Extensive experimental results demonstrate the superiority of our approach, termed ChronoDepth, over existing alternatives, particularly in terms of the temporal consistency of the estimated depth. Additionally, we highlight the benefits of more consistent video depth in two practical applications: depth-conditioned video generation and novel view synthesis. Our project page is available at https://jhaoshao.github.io/ChronoDepth/.

Related papers

• Depth Any Video with Scalable Synthetic Data [98.42356740981839]
  We develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse synthetic environments. We leverage the powerful priors of generative video diffusion models to handle real-world videos effectively. Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.
  arXiv  Detail & Related papers  (2024-10-14T17:59:46Z)
• FIFO-Diffusion: Generating Infinite Videos from Text without Training [44.65468310143439]
  FIFO-Diffusion is conceptually capable of generating infinitely long videos without additional training. Our method dequeues a fully denoised frame at the head while enqueuing a new random noise frame at the tail. We have demonstrated the promising results and effectiveness of the proposed methods on existing text-to-video generation baselines.
  arXiv  Detail & Related papers  (2024-05-19T07:48:41Z)
• NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation [58.21817572577012]
  Video depth estimation aims to infer temporally consistent depth. We introduce NVDS+ that stabilizes inconsistent depth estimated by various single-image models in a plug-and-play manner. We also elaborate a large-scale Video Depth in the Wild dataset, which contains 14,203 videos with over two million frames.
  arXiv  Detail & Related papers  (2023-07-17T17:57:01Z)
• Globally Consistent Video Depth and Pose Estimation with Efficient Test-Time Training [15.46056322267856]
  We present GCVD, a globally consistent method for learning-based video structure from motion (SfM) GCVD integrates a compact pose graph into the CNN-based optimization to achieve globally consistent from an effective selection mechanism. Experimental results show that GCVD outperforms the state-of-the-art methods on both depth and pose estimation.
  arXiv  Detail & Related papers  (2022-08-04T15:12:03Z)
• Controllable Augmentations for Video Representation Learning [34.79719112810065]
  We propose a framework to jointly utilize local clips and global videos to learn from detailed region-level correspondence as well as minimization general long-term temporal relations. Our framework is superior on three video benchmarks in action recognition and video retrieval, capturing more accurate temporal dynamics.
  arXiv  Detail & Related papers  (2022-03-30T19:34:32Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• Deep Video Prior for Video Consistency and Propagation [58.250209011891904]
  We present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly instead of a large dataset. We show that temporal consistency can be achieved by training a convolutional neural network on a video with Deep Video Prior.
  arXiv  Detail & Related papers  (2022-01-27T16:38:52Z)
• ViPNAS: Efficient Video Pose Estimation via Neural Architecture Search [94.90294600817215]
  We propose a novel neural architecture search (NAS) method, termed ViPNAS, to search networks in both spatial and temporal levels for fast online video pose estimation. In the spatial level, we carefully design the search space with five different dimensions including network depth, width, kernel size, group number, and attentions. In the temporal level, we search from a series of temporal feature fusions to optimize the total accuracy and speed across multiple video frames.
  arXiv  Detail & Related papers  (2021-05-21T06:36:40Z)
• Beyond Short Clips: End-to-End Video-Level Learning with Collaborative Memories [56.91664227337115]
  We introduce a collaborative memory mechanism that encodes information across multiple sampled clips of a video at each training iteration. This enables the learning of long-range dependencies beyond a single clip. Our proposed framework is end-to-end trainable and significantly improves the accuracy of video classification at a negligible computational overhead.
  arXiv  Detail & Related papers  (2021-04-02T18:59:09Z)
• Blind Video Temporal Consistency via Deep Video Prior [61.062900556483164]
  We present a novel and general approach for blind video temporal consistency. Our method is only trained on a pair of original and processed videos directly. We show that temporal consistency can be achieved by training a convolutional network on a video with the Deep Video Prior.
  arXiv  Detail & Related papers  (2020-10-22T16:19:20Z)
• Efficient Semantic Video Segmentation with Per-frame Inference [117.97423110566963]
  In this work, we process efficient semantic video segmentation in a per-frame fashion during the inference process. We employ compact models for real-time execution. To narrow the performance gap between compact models and large models, new knowledge distillation methods are designed.
  arXiv  Detail & Related papers  (2020-02-26T12:24:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.