Human Pose and Shape Estimation from Single Polarization Images
- URL: http://arxiv.org/abs/2108.06834v1
- Date: Sun, 15 Aug 2021 22:56:18 GMT
- Title: Human Pose and Shape Estimation from Single Polarization Images
- Authors: Shihao Zou, Xinxin Zuo, Sen Wang, Yiming Qian, Chuan Guo, Wei Ji,
Jingjing Li, Minglun Gong, Li Cheng
- Abstract summary: We attempt to estimate human pose and shape from single polarization images by leveraging the polarization-induced geometric cues.
A dedicated dataset (PHSPD) is constructed, consisting of over 500K frames with accurate pose and shape annotations.
It suggests polarization camera as a promising alternative to the more conventional RGB camera for human pose and shape estimation.
- Score: 45.24275141578927
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper focuses on a new problem of estimating human pose and shape from
single polarization images. Polarization camera is known to be able to capture
the polarization of reflected lights that preserves rich geometric cues of an
object surface. Inspired by the recent applications in surface normal
reconstruction from polarization images, in this paper, we attempt to estimate
human pose and shape from single polarization images by leveraging the
polarization-induced geometric cues. A dedicated two-stage pipeline is
proposed: given a single polarization image, stage one (Polar2Normal) focuses
on the fine detailed human body surface normal estimation; stage two
(Polar2Shape) then reconstructs clothed human shape from the polarization image
and the estimated surface normal. To empirically validate our approach, a
dedicated dataset (PHSPD) is constructed, consisting of over 500K frames with
accurate pose and shape annotations. Empirical evaluations on this real-world
dataset as well as a synthetic dataset, SURREAL, demonstrate the effectiveness
of our approach. It suggests polarization camera as a promising alternative to
the more conventional RGB camera for human pose and shape estimation.
Related papers
- NeRSP: Neural 3D Reconstruction for Reflective Objects with Sparse Polarized Images [62.752710734332894]
NeRSP is a Neural 3D reconstruction technique for Reflective surfaces with Sparse Polarized images.
We derive photometric and geometric cues from the polarimetric image formation model and multiview azimuth consistency.
We achieve the state-of-the-art surface reconstruction results with only 6 views as input.
arXiv Detail & Related papers (2024-06-11T09:53:18Z) - S2P3: Self-Supervised Polarimetric Pose Prediction [55.43547228561919]
This paper proposes the first self-supervised 6D object pose prediction from multimodal RGB+polarimetric images.
The novel training paradigm comprises 1) a physical model to extract geometric information of polarized light, 2) a teacher-student knowledge distillation scheme and 3) a self-supervised loss formulation through differentiable constraints.
arXiv Detail & Related papers (2023-12-02T10:46:40Z) - Polarimetric Information for Multi-Modal 6D Pose Estimation of
Photometrically Challenging Objects with Limited Data [51.95347650131366]
6D pose estimation pipelines that rely on RGB-only or RGB-D data show limitations for photometrically challenging objects.
A supervised learning-based method utilising complementary polarisation information is proposed to overcome such limitations.
arXiv Detail & Related papers (2023-08-21T10:56:00Z) - Polarimetric Multi-View Inverse Rendering [13.391866136230165]
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) and the degree of polarization (DoP) of reflected light are related to an object's surface normal.
We propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that effectively exploits geometric, photometric, and polarimetric cues extracted from input multi-view color-polarization images.
arXiv Detail & Related papers (2022-12-24T12:12:12Z) - PolarFormer: Multi-camera 3D Object Detection with Polar Transformers [93.49713023975727]
3D object detection in autonomous driving aims to reason "what" and "where" the objects of interest present in a 3D world.
Existing methods often adopt the canonical Cartesian coordinate system with perpendicular axis.
We propose a new Polar Transformer (PolarFormer) for more accurate 3D object detection in the bird's-eye-view (BEV) taking as input only multi-camera 2D images.
arXiv Detail & Related papers (2022-06-30T16:32:48Z) - Deep Polarization Imaging for 3D shape and SVBRDF Acquisition [7.86578678811226]
We present a novel method for efficient acquisition of shape and spatially varying reflectance of 3D objects using polarization cues.
Unlike previous works that have exploited polarization to estimate material or object appearance under certain constraints, we lift such restrictions by coupling polarization imaging with deep learning.
We demonstrate our approach to achieve superior results compared to recent works employing deep learning in conjunction with flash illumination.
arXiv Detail & Related papers (2021-05-06T17:58:43Z) - Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-View
Geometry [62.29762409558553]
Epipolar constraints are at the core of feature matching and depth estimation in multi-person 3D human pose estimation methods.
Despite the satisfactory performance of this formulation in sparser crowd scenes, its effectiveness is frequently challenged under denser crowd circumstances.
In this paper, we depart from the multi-person 3D pose estimation formulation, and instead reformulate it as crowd pose estimation.
arXiv Detail & Related papers (2020-07-21T17:59:36Z) - 3D Human Shape Reconstruction from a Polarization Image [34.240256720930155]
This paper tackles the problem of estimating 3D body shape of clothed humans from single polarized 2D images.
A dedicated two-stage deep learning approach, SfP, is proposed.
arXiv Detail & Related papers (2020-07-17T22:36:02Z) - Polarimetric Multi-View Inverse Rendering [13.391866136230165]
A polarization camera has great potential for 3D reconstruction since the angle of polarization (AoP) of reflected light is related to an object's surface normal.
We propose a novel 3D reconstruction method called Polarimetric Multi-View Inverse Rendering (Polarimetric MVIR) that exploits geometric, photometric, and polarimetric cues extracted from input multi-view color polarization images.
arXiv Detail & Related papers (2020-07-17T09:00:20Z) - Polarization Human Shape and Pose Dataset [35.156049015251035]
Polarization images are known to capture polarized reflected lights that preserve rich geometric cues of an object.
Inspired by the recent breakthroughs in human shape estimation from a single color image, we attempt to investigate the new question of whether the geometric cues from polarization camera could be leveraged in estimating detailed human body shapes.
arXiv Detail & Related papers (2020-04-30T15:58:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.