FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution

• URL: http://arxiv.org/abs/2504.07093v1
• Date: Wed, 09 Apr 2025 17:59:31 GMT
• Title: FlashDepth: Real-time Streaming Video Depth Estimation at 2K Resolution
• Authors: Gene Chou, Wenqi Xian, Guandao Yang, Mohamed Abdelfattah, Bharath Hariharan, Noah Snavely, Ning Yu, Paul Debevec,
• Abstract summary: A versatile video depth estimation model should (1) be accurate across frames, (2) produce high-resolution depth maps, and (3) support real-time streaming.<n>We propose FlashDepth, a method that satisfies all three requirements, performing depth estimation on a 2044x1148 streaming video at 24 FPS.
• Score: 50.55876151973996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A versatile video depth estimation model should (1) be accurate and consistent across frames, (2) produce high-resolution depth maps, and (3) support real-time streaming. We propose FlashDepth, a method that satisfies all three requirements, performing depth estimation on a 2044x1148 streaming video at 24 FPS. We show that, with careful modifications to pretrained single-image depth models, these capabilities are enabled with relatively little data and training. We evaluate our approach across multiple unseen datasets against state-of-the-art depth models, and find that ours outperforms them in terms of boundary sharpness and speed by a significant margin, while maintaining competitive accuracy. We hope our model will enable various applications that require high-resolution depth, such as video editing, and online decision-making, such as robotics.

Related papers

• Video Depth Anything: Consistent Depth Estimation for Super-Long Videos [60.857723250653976]
  We propose Video Depth Anything for high-quality, consistent depth estimation in super-long videos.<n>Our model is trained on a joint dataset of video depth and unlabeled images, similar to Depth Anything V2.<n>Our approach sets a new state-of-the-art in zero-shot video depth estimation.
  arXiv  Detail & Related papers  (2025-01-21T18:53:30Z)
• Align3R: Aligned Monocular Depth Estimation for Dynamic Videos [50.28715151619659]
  We propose a novel video-depth estimation method called Align3R to estimate temporal consistent depth maps for a dynamic video. Our key idea is to utilize the recent DUSt3R model to align estimated monocular depth maps of different timesteps. Experiments demonstrate that Align3R estimates consistent video depth and camera poses for a monocular video with superior performance than baseline methods.
  arXiv  Detail & Related papers  (2024-12-04T07:09:59Z)
• Video Depth without Video Models [34.11454612504574]
  Video depth estimation lifts monocular video clips to 3D by inferring dense depth at every frame. We show how to turn a single-image latent diffusion model (LDM) into a state-of-the-art video depth estimator. Our model, which we call RollingDepth, has two main ingredients: (i) a multi-frame depth estimator that is derived from a single-image LDM and maps very short video snippets to depth snippets.
  arXiv  Detail & Related papers  (2024-11-28T14:50:14Z)
• Depth Any Video with Scalable Synthetic Data [98.42356740981839]
  We develop a scalable synthetic data pipeline, capturing real-time video depth data from diverse virtual environments.<n>We leverage the powerful priors of generative video diffusion models to handle real-world videos effectively.<n>Our model outperforms all previous generative depth models in terms of spatial accuracy and temporal consistency.
  arXiv  Detail & Related papers  (2024-10-14T17:59:46Z)
• NVDS+: Towards Efficient and Versatile Neural Stabilizer for Video Depth Estimation [58.21817572577012]
  Video depth estimation aims to infer temporally consistent depth. We introduce NVDS+ that stabilizes inconsistent depth estimated by various single-image models in a plug-and-play manner. We also elaborate a large-scale Video Depth in the Wild dataset, which contains 14,203 videos with over two million frames.
  arXiv  Detail & Related papers  (2023-07-17T17:57:01Z)
• Shakes on a Plane: Unsupervised Depth Estimation from Unstabilized Photography [54.36608424943729]
  We show that in a ''long-burst'', forty-two 12-megapixel RAW frames captured in a two-second sequence, there is enough parallax information from natural hand tremor alone to recover high-quality scene depth. We devise a test-time optimization approach that fits a neural RGB-D representation to long-burst data and simultaneously estimates scene depth and camera motion.
  arXiv  Detail & Related papers  (2022-12-22T18:54:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.