MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos

• URL: http://arxiv.org/abs/2412.04463v2
• Date: Fri, 06 Dec 2024 19:15:46 GMT
• Title: MegaSaM: Accurate, Fast, and Robust Structure and Motion from Casual Dynamic Videos
• Authors: Zhengqi Li, Richard Tucker, Forrester Cole, Qianqian Wang, Linyi Jin, Vickie Ye, Angjoo Kanazawa, Aleksander Holynski, Noah Snavely,
• Abstract summary: We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes. Our system is significantly more accurate and robust at camera pose and depth estimation when compared with prior and concurrent work.
• Score: 104.1338295060383
• License:
• Abstract: We present a system that allows for accurate, fast, and robust estimation of camera parameters and depth maps from casual monocular videos of dynamic scenes. Most conventional structure from motion and monocular SLAM techniques assume input videos that feature predominantly static scenes with large amounts of parallax. Such methods tend to produce erroneous estimates in the absence of these conditions. Recent neural network-based approaches attempt to overcome these challenges; however, such methods are either computationally expensive or brittle when run on dynamic videos with uncontrolled camera motion or unknown field of view. We demonstrate the surprising effectiveness of a deep visual SLAM framework: with careful modifications to its training and inference schemes, this system can scale to real-world videos of complex dynamic scenes with unconstrained camera paths, including videos with little camera parallax. Extensive experiments on both synthetic and real videos demonstrate that our system is significantly more accurate and robust at camera pose and depth estimation when compared with prior and concurrent work, with faster or comparable running times. See interactive results on our project page: https://mega-sam.github.io/

Related papers

• Cavia: Camera-controllable Multi-view Video Diffusion with View-Integrated Attention [62.2447324481159]
  Cavia is a novel framework for camera-controllable, multi-view video generation. Our framework extends the spatial and temporal attention modules, improving both viewpoint and temporal consistency. Cavia is the first of its kind that allows the user to specify distinct camera motion while obtaining object motion.
  arXiv  Detail & Related papers  (2024-10-14T17:46:32Z)
• Generative Camera Dolly: Extreme Monocular Dynamic Novel View Synthesis [43.02778060969546]
  We propose a controllable monocular dynamic view synthesis pipeline. Our model does not require depth as input, and does not explicitly model 3D scene geometry. We believe our framework can potentially unlock powerful applications in rich dynamic scene understanding, perception for robotics, and interactive 3D video viewing experiences for virtual reality.
  arXiv  Detail & Related papers  (2024-05-23T17:59:52Z)
• Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image [59.18564636990079]
  We study the problem of synthesizing a long-term dynamic video from only a single image. Existing methods either hallucinate inconsistent perpetual views or struggle with long camera trajectories. We present Make-It-4D, a novel method that can generate a consistent long-term dynamic video from a single image.
  arXiv  Detail & Related papers  (2023-08-20T12:53:50Z)
• Decoupling Dynamic Monocular Videos for Dynamic View Synthesis [50.93409250217699]
  We tackle the challenge of dynamic view synthesis from dynamic monocular videos in an unsupervised fashion. Specifically, we decouple the motion of the dynamic objects into object motion and camera motion, respectively regularized by proposed unsupervised surface consistency and patch-based multi-view constraints.
  arXiv  Detail & Related papers  (2023-04-04T11:25:44Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• Low Light Video Enhancement by Learning on Static Videos with Cross-Frame Attention [10.119600046984088]
  We develop a deep learning method for low light video enhancement by training a model on static videos. Existing methods operate frame by frame and do not exploit the relationships among neighbouring frames. We show that our method outperforms other state-of-the-art video enhancement algorithms when trained only on static videos.
  arXiv  Detail & Related papers  (2022-10-09T15:49:46Z)
• ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild [57.37891682117178]
  We present a robust dense indirect structure-from-motion method for videos that is based on dense correspondence from pairwise optical flow. A novel neural network architecture is proposed for processing irregular point trajectory data. Experiments on MPI Sintel dataset show that our system produces significantly more accurate camera trajectories.
  arXiv  Detail & Related papers  (2022-07-19T09:19:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.