Learning non-rigid surface reconstruction from spatio-temporal image patches

• URL: http://arxiv.org/abs/2006.10841v1
• Date: Thu, 18 Jun 2020 20:25:15 GMT
• Title: Learning non-rigid surface reconstruction from spatio-temporal image patches
• Authors: Matteo Pedone, Abdelrahman Mostafa, Janne heikkil\"a
• Abstract summary: We present a method to reconstruct a dense-temporal depth map of a deformable object from a video sequence. The estimation of depth is performed locally on non-temporal patches of the video, and the full depth video of entire shape is recovered by combining them together. We tested our method on both synthetic and Kinect data and experimentally observed that the reconstruction error is significantly lower than the one obtained using other approaches like conventional non-rigid structure.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a method to reconstruct a dense spatio-temporal depth map of a non-rigidly deformable object directly from a video sequence. The estimation of depth is performed locally on spatio-temporal patches of the video, and then the full depth video of the entire shape is recovered by combining them together. Since the geometric complexity of a local spatio-temporal patch of a deforming non-rigid object is often simple enough to be faithfully represented with a parametric model, we artificially generate a database of small deforming rectangular meshes rendered with different material properties and light conditions, along with their corresponding depth videos, and use such data to train a convolutional neural network. We tested our method on both synthetic and Kinect data and experimentally observed that the reconstruction error is significantly lower than the one obtained using other approaches like conventional non-rigid structure from motion.

Related papers

• High-Quality Mesh Blendshape Generation from Face Videos via Neural Inverse Rendering [15.009484906668737]
  We introduce a novel technique that reconstructs mesh-based blendshape rigs from single or sparse multi-view videos. Experiments demonstrate that, with the flexible input of single or sparse multi-view videos, we reconstruct personalized high-fidelity blendshapes.
  arXiv  Detail & Related papers  (2024-01-16T14:41:31Z)
• SceNeRFlow: Time-Consistent Reconstruction of General Dynamic Scenes [75.9110646062442]
  We propose SceNeRFlow to reconstruct a general, non-rigid scene in a time-consistent manner. Our method takes multi-view RGB videos and background images from static cameras with known camera parameters as input. We show experimentally that, unlike prior work that only handles small motion, our method enables the reconstruction of studio-scale motions.
  arXiv  Detail & Related papers  (2023-08-16T09:50:35Z)
• Object-Centric Learning for Real-World Videos by Predicting Temporal Feature Similarities [30.564704737585558]
  We propose a novel way to use pre-trained features in the form of a temporal feature similarity loss. This loss encodes semantic and temporal correlations between image patches and is a natural way to introduce a motion bias for object discovery. We demonstrate that this loss leads to state-of-the-art performance on the challenging synthetic MOVi datasets.
  arXiv  Detail & Related papers  (2023-06-07T23:18:14Z)
• Learning Topology from Synthetic Data for Unsupervised Depth Completion [66.26787962258346]
  We present a method for inferring dense depth maps from images and sparse depth measurements. We learn the association of sparse point clouds with dense natural shapes, and using the image as evidence to validate the predicted depth map.
  arXiv  Detail & Related papers  (2021-06-06T00:21:12Z)
• Multi-view Depth Estimation using Epipolar Spatio-Temporal Networks [87.50632573601283]
  We present a novel method for multi-view depth estimation from a single video. Our method achieves temporally coherent depth estimation results by using a novel Epipolar Spatio-Temporal (EST) transformer. To reduce the computational cost, inspired by recent Mixture-of-Experts models, we design a compact hybrid network.
  arXiv  Detail & Related papers  (2020-11-26T04:04:21Z)
• Category Level Object Pose Estimation via Neural Analysis-by-Synthesis [64.14028598360741]
  In this paper we combine a gradient-based fitting procedure with a parametric neural image synthesis module. The image synthesis network is designed to efficiently span the pose configuration space. We experimentally show that the method can recover orientation of objects with high accuracy from 2D images alone.
  arXiv  Detail & Related papers  (2020-08-18T20:30:47Z)
• A Plug-and-play Scheme to Adapt Image Saliency Deep Model for Video Data [54.198279280967185]
  This paper proposes a novel plug-and-play scheme to weakly retrain a pretrained image saliency deep model for video data. Our method is simple yet effective for adapting any off-the-shelf pre-trained image saliency deep model to obtain high-quality video saliency detection.
  arXiv  Detail & Related papers  (2020-08-02T13:23:14Z)
• Consistent Video Depth Estimation [57.712779457632024]
  We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion.
  arXiv  Detail & Related papers  (2020-04-30T17:59:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.