STATIC : Surface Temporal Affine for TIme Consistency in Video Monocular Depth Estimation

• URL: http://arxiv.org/abs/2412.01090v1
• Date: Mon, 02 Dec 2024 03:53:33 GMT
• Title: STATIC : Surface Temporal Affine for TIme Consistency in Video Monocular Depth Estimation
• Authors: Sunghun Yang, Minhyeok Lee, Suhwan Cho, Jungho Lee, Sangyoun Lee,
• Abstract summary: Video monocular depth estimation is essential for applications such as autonomous driving, AR/VR, and robotics. Recent transformer-based single-image monocular depth estimation models perform well on single images but struggle with depth consistency across video frames. We propose STATIC, a novel model that learns temporal consistency in static and dynamic area without additional information.
• Score: 14.635179908525389
• License:
• Abstract: Video monocular depth estimation is essential for applications such as autonomous driving, AR/VR, and robotics. Recent transformer-based single-image monocular depth estimation models perform well on single images but struggle with depth consistency across video frames. Traditional methods aim to improve temporal consistency using multi-frame temporal modules or prior information like optical flow and camera parameters. However, these approaches face issues such as high memory use, reduced performance with dynamic or irregular motion, and limited motion understanding. We propose STATIC, a novel model that independently learns temporal consistency in static and dynamic area without additional information. A difference mask from surface normals identifies static and dynamic area by measuring directional variance. For static area, the Masked Static (MS) module enhances temporal consistency by focusing on stable regions. For dynamic area, the Surface Normal Similarity (SNS) module aligns areas and enhances temporal consistency by measuring feature similarity between frames. A final refinement integrates the independently learned static and dynamic area, enabling STATIC to achieve temporal consistency across the entire sequence. Our method achieves state-of-the-art video depth estimation on the KITTI and NYUv2 datasets without additional information.

Related papers

• DATAP-SfM: Dynamic-Aware Tracking Any Point for Robust Structure from Motion in the Wild [85.03973683867797]
  This paper proposes a concise, elegant, and robust pipeline to estimate smooth camera trajectories and obtain dense point clouds for casual videos in the wild. We show that the proposed method achieves state-of-the-art performance in terms of camera pose estimation even in complex dynamic challenge scenes.
  arXiv  Detail & Related papers  (2024-11-20T13:01:16Z)
• OmniLocalRF: Omnidirectional Local Radiance Fields from Dynamic Videos [14.965321452764355]
  We introduce a new approach called Omnidirectional Local Radiance Fields (OmniLocalRF) that can render static-only scene views. Our approach combines the principles of local radiance fields with the bidirectional optimization of omnidirectional rays. Our experiments validate that OmniLocalRF outperforms existing methods in both qualitative and quantitative metrics.
  arXiv  Detail & Related papers  (2024-03-31T12:55:05Z)
• EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision [85.17951804790515]
  EmerNeRF is a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes. It simultaneously captures scene geometry, appearance, motion, and semantics via self-bootstrapping. Our method achieves state-of-the-art performance in sensor simulation.
  arXiv  Detail & Related papers  (2023-11-03T17:59:55Z)
• Alignment-free HDR Deghosting with Semantics Consistent Transformer [76.91669741684173]
  High dynamic range imaging aims to retrieve information from multiple low-dynamic range inputs to generate realistic output. Existing methods often focus on the spatial misalignment across input frames caused by the foreground and/or camera motion. We propose a novel alignment-free network with a Semantics Consistent Transformer (SCTNet) with both spatial and channel attention modules.
  arXiv  Detail & Related papers  (2023-05-29T15:03:23Z)
• Robust Dynamic Radiance Fields [79.43526586134163]
  Dynamic radiance field reconstruction methods aim to model the time-varying structure and appearance of a dynamic scene. Existing methods, however, assume that accurate camera poses can be reliably estimated by Structure from Motion (SfM) algorithms. We address this robustness issue by jointly estimating the static and dynamic radiance fields along with the camera parameters.
  arXiv  Detail & Related papers  (2023-01-05T18:59:51Z)
• STVGFormer: Spatio-Temporal Video Grounding with Static-Dynamic Cross-Modal Understanding [68.96574451918458]
  We propose a framework named STVG, which models visual-linguistic dependencies with a static branch and a dynamic branch. Both the static and dynamic branches are designed as cross-modal transformers. Our proposed method achieved 39.6% vIoU and won the first place in the HC-STVG of the Person in Context Challenge.
  arXiv  Detail & Related papers  (2022-07-06T15:48:58Z)
• TSI: Temporal Saliency Integration for Video Action Recognition [32.18535820790586]
  We propose a Temporal Saliency Integration (TSI) block, which mainly contains a Salient Motion Excitation (SME) module and a Cross-scale Temporal Integration (CTI) module. SME aims to highlight the motion-sensitive area through local-global motion modeling. CTI is designed to perform multi-scale temporal modeling through a group of separate 1D convolutions respectively.
  arXiv  Detail & Related papers  (2021-06-02T11:43:49Z)
• TAM: Temporal Adaptive Module for Video Recognition [60.83208364110288]
  temporal adaptive module (bf TAM) generates video-specific temporal kernels based on its own feature map. Experiments on Kinetics-400 and Something-Something datasets demonstrate that our TAM outperforms other temporal modeling methods consistently.
  arXiv  Detail & Related papers  (2020-05-14T08:22:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.