3D Motion Magnification: Visualizing Subtle Motions with Time Varying Radiance Fields

• URL: http://arxiv.org/abs/2308.03757v1
• Date: Mon, 7 Aug 2023 17:59:59 GMT
• Title: 3D Motion Magnification: Visualizing Subtle Motions with Time Varying Radiance Fields
• Authors: Brandon Y. Feng, Hadi Alzayer, Michael Rubinstein, William T. Freeman, Jia-Bin Huang
• Abstract summary: We present a 3D motion magnification method that can magnify subtle motions from scenes captured by a moving camera. We represent the scene with time-varying radiance fields and leverage the Eulerian principle for motion magnification. We evaluate the effectiveness of our method on both synthetic and real-world scenes captured under various camera setups.
• Score: 58.6780687018956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Motion magnification helps us visualize subtle, imperceptible motion. However, prior methods only work for 2D videos captured with a fixed camera. We present a 3D motion magnification method that can magnify subtle motions from scenes captured by a moving camera, while supporting novel view rendering. We represent the scene with time-varying radiance fields and leverage the Eulerian principle for motion magnification to extract and amplify the variation of the embedding of a fixed point over time. We study and validate our proposed principle for 3D motion magnification using both implicit and tri-plane-based radiance fields as our underlying 3D scene representation. We evaluate the effectiveness of our method on both synthetic and real-world scenes captured under various camera setups.

Related papers

• Shape of Motion: 4D Reconstruction from a Single Video [51.04575075620677]
  We introduce a method capable of reconstructing generic dynamic scenes, featuring explicit, full-sequence-long 3D motion. We exploit the low-dimensional structure of 3D motion by representing scene motion with a compact set of SE3 motion bases. Our method achieves state-of-the-art performance for both long-range 3D/2D motion estimation and novel view synthesis on dynamic scenes.
  arXiv  Detail & Related papers  (2024-07-18T17:59:08Z)
• Cinematic Behavior Transfer via NeRF-based Differentiable Filming [63.1622492808519]
  Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections. We first introduce a reverse filming behavior estimation technique. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment.
  arXiv  Detail & Related papers  (2023-11-29T15:56:58Z)
• Joint 3D Shape and Motion Estimation from Rolling Shutter Light-Field Images [2.0277446818410994]
  We propose an approach to address the problem of 3D reconstruction of scenes from a single image captured by a light-field camera equipped with a rolling shutter sensor. Our method leverages the 3D information cues present in the light-field and the motion information provided by the rolling shutter effect. We present a generic model for the imaging process of this sensor and a two-stage algorithm that minimizes the re-projection error.
  arXiv  Detail & Related papers  (2023-11-02T15:08:18Z)
• 3D Cinemagraphy from a Single Image [73.09720823592092]
  We present 3D Cinemagraphy, a new technique that marries 2D image animation with 3D photography. Given a single still image as input, our goal is to generate a video that contains both visual content animation and camera motion.
  arXiv  Detail & Related papers  (2023-03-10T06:08:23Z)
• 3D Moments from Near-Duplicate Photos [67.15199743223332]
  3D Moments is a new computational photography effect. We produce a video that smoothly interpolates the scene motion from the first photo to the second. Our system produces photorealistic space-time videos with motion parallax and scene dynamics.
  arXiv  Detail & Related papers  (2022-05-12T17:56:18Z)
• Motion-from-Blur: 3D Shape and Motion Estimation of Motion-blurred Objects in Videos [115.71874459429381]
  We propose a method for jointly estimating the 3D motion, 3D shape, and appearance of highly motion-blurred objects from a video. Experiments on benchmark datasets demonstrate that our method outperforms previous methods for fast moving object deblurring and 3D reconstruction.
  arXiv  Detail & Related papers  (2021-11-29T11:25:14Z)
• Visual Odometry with an Event Camera Using Continuous Ray Warping and Volumetric Contrast Maximization [31.627936023222052]
  We present a new solution to tracking and mapping with an event camera. The motion of the camera contains both rotation and translation, and the displacements happen in an arbitrarily structured environment. We introduce a new solution to this problem by performing contrast in 3D. The practical validity of our approach is supported by an application to AGV motion estimation and 3D reconstruction with a single vehicle-mounted event camera.
  arXiv  Detail & Related papers  (2021-07-07T04:32:57Z)
• Synergetic Reconstruction from 2D Pose and 3D Motion for Wide-Space Multi-Person Video Motion Capture in the Wild [3.0015034534260665]
  We propose a markerless motion capture method with accuracy and smoothness from multiple cameras. The proposed method predicts each persons 3D pose and determines bounding box of multi-camera images. We evaluated the proposed method using various datasets and a real sports field.
  arXiv  Detail & Related papers  (2020-01-16T02:14:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.