3D shape reconstruction of semi-transparent worms

• URL: http://arxiv.org/abs/2304.14841v1
• Date: Fri, 28 Apr 2023 13:29:36 GMT
• Title: 3D shape reconstruction of semi-transparent worms
• Authors: Thomas P. Ilett, Omer Yuval, Thomas Ranner, Netta Cohen, David C. Hogg
• Abstract summary: 3D shape reconstruction typically requires identifying object features or textures in multiple images of a subject. Here we overcome these challenges by rendering a candidate shape with adaptive blurring and transparency for comparison with the images. We model the slender Caenorhabditis elegans as a 3D curve using an intrinsic parametrisation that naturally admits biologically-informed constraints and regularisation.
• Score: 0.950214811819847
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: 3D shape reconstruction typically requires identifying object features or textures in multiple images of a subject. This approach is not viable when the subject is semi-transparent and moving in and out of focus. Here we overcome these challenges by rendering a candidate shape with adaptive blurring and transparency for comparison with the images. We use the microscopic nematode Caenorhabditis elegans as a case study as it freely explores a 3D complex fluid with constantly changing optical properties. We model the slender worm as a 3D curve using an intrinsic parametrisation that naturally admits biologically-informed constraints and regularisation. To account for the changing optics we develop a novel differentiable renderer to construct images from 2D projections and compare against raw images to generate a pixel-wise error to jointly update the curve, camera and renderer parameters using gradient descent. The method is robust to interference such as bubbles and dirt trapped in the fluid, stays consistent through complex sequences of postures, recovers reliable estimates from blurry images and provides a significant improvement on previous attempts to track C. elegans in 3D. Our results demonstrate the potential of direct approaches to shape estimation in complex physical environments in the absence of ground-truth data.

Related papers

• Decaf: Monocular Deformation Capture for Face and Hand Interactions [77.75726740605748]
  This paper introduces the first method that allows tracking human hands interacting with human faces in 3D from single monocular RGB videos. We model hands as articulated objects inducing non-rigid face deformations during an active interaction. Our method relies on a new hand-face motion and interaction capture dataset with realistic face deformations acquired with a markerless multi-view camera system.
  arXiv  Detail & Related papers  (2023-09-28T17:59:51Z)
• ARTIC3D: Learning Robust Articulated 3D Shapes from Noisy Web Image Collections [71.46546520120162]
  Estimating 3D articulated shapes like animal bodies from monocular images is inherently challenging. We propose ARTIC3D, a self-supervised framework to reconstruct per-instance 3D shapes from a sparse image collection in-the-wild. We produce realistic animations by fine-tuning the rendered shape and texture under rigid part transformations.
  arXiv  Detail & Related papers  (2023-06-07T17:47:50Z)
• MoDA: Modeling Deformable 3D Objects from Casual Videos [84.29654142118018]
  We propose neural dual quaternion blend skinning (NeuDBS) to achieve 3D point deformation without skin-collapsing artifacts. In the endeavor to register 2D pixels across different frames, we establish a correspondence between canonical feature embeddings that encodes 3D points within the canonical space. Our approach can reconstruct 3D models for humans and animals with better qualitative and quantitative performance than state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-04-17T13:49:04Z)
• Multi-Person 3D Pose and Shape Estimation via Inverse Kinematics and Refinement [5.655207244072081]
  Estimating 3D poses and shapes in the form of meshes from monocular RGB images is challenging. We propose a coarse-to-fine pipeline that benefits from 1) inverse kinematics from the occlusion-robust 3D skeleton estimation. We demonstrate the effectiveness of our method, outperforming state-of-the-arts on 3DPW, MuPoTS and AGORA datasets.
  arXiv  Detail & Related papers  (2022-10-24T18:29:06Z)
• {\phi}-SfT: Shape-from-Template with a Physics-Based Deformation Model [69.27632025495512]
  Shape-from-Template (SfT) methods estimate 3D surface deformations from a single monocular RGB camera. This paper proposes a new SfT approach explaining 2D observations through physical simulations.
  arXiv  Detail & Related papers  (2022-03-22T17:59:57Z)
• A Shading-Guided Generative Implicit Model for Shape-Accurate 3D-Aware Image Synthesis [163.96778522283967]
  We propose a shading-guided generative implicit model that is able to learn a starkly improved shape representation. An accurate 3D shape should also yield a realistic rendering under different lighting conditions. Our experiments on multiple datasets show that the proposed approach achieves photorealistic 3D-aware image synthesis.
  arXiv  Detail & Related papers  (2021-10-29T10:53:12Z)
• Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects [115.71874459429381]
  We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from a single motion-blurred image. While previous approaches address the deblurring problem only in the 2D image domain, our proposed rigorous modeling of all object properties in the 3D domain enables the correct description of arbitrary object motion.
  arXiv  Detail & Related papers  (2021-06-16T13:18:08Z)
• Using Adaptive Gradient for Texture Learning in Single-View 3D Reconstruction [0.0]
  Learning-based approaches for 3D model reconstruction have attracted attention owing to its modern applications. We present a novel sampling algorithm by optimizing the gradient of predicted coordinates based on the variance on the sampling image. We also adopt Frechet Inception Distance (FID) to form a loss function in learning, which helps bridging the gap between rendered images and input images.
  arXiv  Detail & Related papers  (2021-04-29T07:52:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.