Human from Blur: Human Pose Tracking from Blurry Images

• URL: http://arxiv.org/abs/2303.17209v3
• Date: Mon, 25 Sep 2023 23:31:49 GMT
• Title: Human from Blur: Human Pose Tracking from Blurry Images
• Authors: Yiming Zhao, Denys Rozumnyi, Jie Song, Otmar Hilliges, Marc Pollefeys, Martin R. Oswald
• Abstract summary: We propose a method to estimate 3D human poses from substantially blurred images. Key idea is to tackle the inverse problem of image deblurring by modeling the forward problem with a 3D human model, a texture map, and a sequence of poses to describe human motion. Using a differentiable step, we can solve the inverse problem by backpropagating the pixel-wise reprojection error to recover the best human motion representation.
• Score: 89.65036443997103
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a method to estimate 3D human poses from substantially blurred images. The key idea is to tackle the inverse problem of image deblurring by modeling the forward problem with a 3D human model, a texture map, and a sequence of poses to describe human motion. The blurring process is then modeled by a temporal image aggregation step. Using a differentiable renderer, we can solve the inverse problem by backpropagating the pixel-wise reprojection error to recover the best human motion representation that explains a single or multiple input images. Since the image reconstruction loss alone is insufficient, we present additional regularization terms. To the best of our knowledge, we present the first method to tackle this problem. Our method consistently outperforms other methods on significantly blurry inputs since they lack one or multiple key functionalities that our method unifies, i.e. image deblurring with sub-frame accuracy and explicit 3D modeling of non-rigid human motion.

Related papers

• Personalized 3D Human Pose and Shape Refinement [19.082329060985455]
  regression-based methods have dominated the field of 3D human pose and shape estimation. We propose to construct dense correspondences between initial human model estimates and the corresponding images. We show that our approach not only consistently leads to better image-model alignment, but also to improved 3D accuracy.
  arXiv  Detail & Related papers  (2024-03-18T10:13:53Z)
• Score-Guided Diffusion for 3D Human Recovery [10.562998991986102]
  We present Score-Guided Human Mesh Recovery (ScoreHMR), an approach for solving inverse problems for 3D human pose and shape reconstruction. ScoreHMR mimics model fitting approaches, but alignment with the image observation is achieved through score guidance in the latent space of a diffusion model. We evaluate our approach on three settings/applications: (i) single-frame model fitting; (ii) reconstruction from multiple uncalibrated views; (iii) reconstructing humans in video sequences.
  arXiv  Detail & Related papers  (2024-03-14T17:56:14Z)
• Synthesizing Moving People with 3D Control [88.68284137105654]
  We present a diffusion model-based framework for animating people from a single image for a given target 3D motion sequence. For the first part, we learn an in-filling diffusion model to hallucinate unseen parts of a person given a single image. Second, we develop a diffusion-based rendering pipeline, which is controlled by 3D human poses.
  arXiv  Detail & Related papers  (2024-01-19T18:59:11Z)
• SiTH: Single-view Textured Human Reconstruction with Image-Conditioned Diffusion [35.73448283467723]
  SiTH is a novel pipeline that integrates an image-conditioned diffusion model into a 3D mesh reconstruction workflow. We employ a powerful generative diffusion model to hallucinate unseen back-view appearance based on the input images. For the latter, we leverage skinned body meshes as guidance to recover full-body texture meshes from the input and back-view images.
  arXiv  Detail & Related papers  (2023-11-27T14:22:07Z)
• Refining 3D Human Texture Estimation from a Single Image [3.8761064607384195]
  Estimating 3D human texture from a single image is essential in graphics and vision. We propose a framework that adaptively samples the input by a deformable convolution where offsets are learned via a deep neural network.
  arXiv  Detail & Related papers  (2023-03-06T19:53:50Z)
• Decoupling Human and Camera Motion from Videos in the Wild [67.39432972193929]
  We propose a method to reconstruct global human trajectories from videos in the wild. Our method decouples the camera and human motion, which allows us to place people in the same world coordinate frame.
  arXiv  Detail & Related papers  (2023-02-24T18:59:15Z)
• Shape from Blur: Recovering Textured 3D Shape and Motion of Fast Moving Objects [115.71874459429381]
  We address the novel task of jointly reconstructing the 3D shape, texture, and motion of an object from a single motion-blurred image. While previous approaches address the deblurring problem only in the 2D image domain, our proposed rigorous modeling of all object properties in the 3D domain enables the correct description of arbitrary object motion.
  arXiv  Detail & Related papers  (2021-06-16T13:18:08Z)
• Single Image Non-uniform Blur Kernel Estimation via Adaptive Basis Decomposition [1.854931308524932]
  We propose a general, non-parametric model for dense non-uniform motion blur estimation. We show that our method overcomes the limitations of existing non-uniform motion blur estimation.
  arXiv  Detail & Related papers  (2021-02-01T18:02:31Z)
• Coherent Reconstruction of Multiple Humans from a Single Image [68.3319089392548]
  In this work, we address the problem of multi-person 3D pose estimation from a single image. A typical regression approach in the top-down setting of this problem would first detect all humans and then reconstruct each one of them independently. Our goal is to train a single network that learns to avoid these problems and generate a coherent 3D reconstruction of all the humans in the scene.
  arXiv  Detail & Related papers  (2020-06-15T17:51:45Z)
• Chained Representation Cycling: Learning to Estimate 3D Human Pose and Shape by Cycling Between Representations [73.11883464562895]
  We propose a new architecture that facilitates unsupervised, or lightly supervised, learning. We demonstrate the method by learning 3D human pose and shape from un-paired and un-annotated images. While we present results for modeling humans, our formulation is general and can be applied to other vision problems.
  arXiv  Detail & Related papers  (2020-01-06T14:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.