RoMo: Robust Motion Segmentation Improves Structure from Motion

• URL: http://arxiv.org/abs/2411.18650v1
• Date: Wed, 27 Nov 2024 01:09:56 GMT
• Title: RoMo: Robust Motion Segmentation Improves Structure from Motion
• Authors: Lily Goli, Sara Sabour, Mark Matthews, Marcus Brubaker, Dmitry Lagun, Alec Jacobson, David J. Fleet, Saurabh Saxena, Andrea Tagliasacchi,
• Abstract summary: We propose a novel approach to video-based motion segmentation to identify the components of a scene that are moving w.r.t. a fixed world frame. Our simple but effective iterative method, RoMo, combines optical flow and epipolar cues with a pre-trained video segmentation model. More importantly, the combination of an off-the-shelf SfM pipeline with our segmentation masks establishes a new state-of-the-art on camera calibration for scenes with dynamic content, outperforming existing methods by a substantial margin.
• Score: 46.77236343300953
• License:
• Abstract: There has been extensive progress in the reconstruction and generation of 4D scenes from monocular casually-captured video. While these tasks rely heavily on known camera poses, the problem of finding such poses using structure-from-motion (SfM) often depends on robustly separating static from dynamic parts of a video. The lack of a robust solution to this problem limits the performance of SfM camera-calibration pipelines. We propose a novel approach to video-based motion segmentation to identify the components of a scene that are moving w.r.t. a fixed world frame. Our simple but effective iterative method, RoMo, combines optical flow and epipolar cues with a pre-trained video segmentation model. It outperforms unsupervised baselines for motion segmentation as well as supervised baselines trained from synthetic data. More importantly, the combination of an off-the-shelf SfM pipeline with our segmentation masks establishes a new state-of-the-art on camera calibration for scenes with dynamic content, outperforming existing methods by a substantial margin.

Related papers

• MCDS-VSS: Moving Camera Dynamic Scene Video Semantic Segmentation by Filtering with Self-Supervised Geometry and Motion [17.50161162624179]
  Self-driving cars rely on reliable semantic environment perception for decision making. We propose MCDS-VSS, a structured filter model that learns in a self-supervised manner to estimate scene geometry and ego-motion of the camera. Our model parses automotive scenes into multiple interpretable representations such as scene geometry, ego-motion, and object motion.
  arXiv  Detail & Related papers  (2024-05-30T10:33:14Z)
• Out of the Room: Generalizing Event-Based Dynamic Motion Segmentation for Complex Scenes [10.936350433952668]
  Rapid and reliable identification of dynamic scene parts, also known as motion segmentation, is a key challenge for mobile sensors. Event cameras have the potential to overcome these limitations, but corresponding methods have only been demonstrated in smaller-scale indoor environments. This work presents an event-based method for class-agnostic motion segmentation that can successfully be deployed across complex large-scale outdoor environments too.
  arXiv  Detail & Related papers  (2024-03-07T14:59:34Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• InstMove: Instance Motion for Object-centric Video Segmentation [70.16915119724757]
  In this work, we study the instance-level motion and present InstMove, which stands for Instance Motion for Object-centric Video. In comparison to pixel-wise motion, InstMove mainly relies on instance-level motion information that is free from image feature embeddings. With only a few lines of code, InstMove can be integrated into current SOTA methods for three different video segmentation tasks.
  arXiv  Detail & Related papers  (2023-03-14T17:58:44Z)
• A Deep Moving-camera Background Model [5.564705758320338]
  We propose a new method for learning Moving-camera Background Models (MCBM) in video analysis videos. DeepMCBM eliminates the problems associated with joint alignment and achieves state-of-the-art results. We demonstrate DeepMCBM's utility on a variety of videos, including ones beyond the scope of other methods.
  arXiv  Detail & Related papers  (2022-09-16T13:36:54Z)
• ParticleSfM: Exploiting Dense Point Trajectories for Localizing Moving Cameras in the Wild [57.37891682117178]
  We present a robust dense indirect structure-from-motion method for videos that is based on dense correspondence from pairwise optical flow. A novel neural network architecture is proposed for processing irregular point trajectory data. Experiments on MPI Sintel dataset show that our system produces significantly more accurate camera trajectories.
  arXiv  Detail & Related papers  (2022-07-19T09:19:45Z)
• NeuralDiff: Segmenting 3D objects that move in egocentric videos [92.95176458079047]
  We study the problem of decomposing the observed 3D scene into a static background and a dynamic foreground. This task is reminiscent of the classic background subtraction problem, but is significantly harder because all parts of the scene, static and dynamic, generate a large apparent motion. In particular, we consider egocentric videos and further separate the dynamic component into objects and the actor that observes and moves them.
  arXiv  Detail & Related papers  (2021-10-19T12:51:35Z)
• Learning to Segment Rigid Motions from Two Frames [72.14906744113125]
  We propose a modular network, motivated by a geometric analysis of what independent object motions can be recovered from an egomotion field. It takes two consecutive frames as input and predicts segmentation masks for the background and multiple rigidly moving objects, which are then parameterized by 3D rigid transformations. Our method achieves state-of-the-art performance for rigid motion segmentation on KITTI and Sintel.
  arXiv  Detail & Related papers  (2021-01-11T04:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.