Multimodal In-bed Pose and Shape Estimation under the Blankets
- URL: http://arxiv.org/abs/2012.06735v1
- Date: Sat, 12 Dec 2020 05:35:23 GMT
- Title: Multimodal In-bed Pose and Shape Estimation under the Blankets
- Authors: Yu Yin, Joseph P. Robinson, Yun Fu
- Abstract summary: We propose a pyramid scheme to fuse different modalities in a way that best leverages the knowledge captured by the multimodal sensors.
We employ an attention-based reconstruction module to generate uncovered modalities, which are further fused to update current estimation.
- Score: 77.12439296395733
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Humans spend vast hours in bed -- about one-third of the lifetime on average.
Besides, a human at rest is vital in many healthcare applications. Typically,
humans are covered by a blanket when resting, for which we propose a multimodal
approach to uncover the subjects so their bodies at rest can be viewed without
the occlusion of the blankets above. We propose a pyramid scheme to effectively
fuse the different modalities in a way that best leverages the knowledge
captured by the multimodal sensors. Specifically, the two most informative
modalities (i.e., depth and infrared images) are first fused to generate good
initial pose and shape estimation. Then pressure map and RGB images are further
fused one by one to refine the result by providing occlusion-invariant
information for the covered part, and accurate shape information for the
uncovered part, respectively. However, even with multimodal data, the task of
detecting human bodies at rest is still very challenging due to the extreme
occlusion of bodies. To further reduce the negative effects of the occlusion
from blankets, we employ an attention-based reconstruction module to generate
uncovered modalities, which are further fused to update current estimation via
a cyclic fashion. Extensive experiments validate the superiority of the
proposed model over others.
Related papers
- DPMesh: Exploiting Diffusion Prior for Occluded Human Mesh Recovery [71.6345505427213]
DPMesh is an innovative framework for occluded human mesh recovery.
It capitalizes on the profound diffusion prior about object structure and spatial relationships embedded in a pre-trained text-to-image diffusion model.
arXiv Detail & Related papers (2024-04-01T18:59:13Z) - AiOS: All-in-One-Stage Expressive Human Pose and Shape Estimation [55.179287851188036]
We introduce a novel all-in-one-stage framework, AiOS, for expressive human pose and shape recovery without an additional human detection step.
We first employ a human token to probe a human location in the image and encode global features for each instance.
Then, we introduce a joint-related token to probe the human joint in the image and encoder a fine-grained local feature.
arXiv Detail & Related papers (2024-03-26T17:59:23Z) - In-Bed Pose Estimation: A Review [8.707107668375906]
In-bed pose estimation can be used to monitor a person's sleep behavior and detect symptoms early for potential disease diagnosis.
Several studies have utilized unimodal and multimodal methods to estimate in-bed human poses.
Our objectives are to show the limitations of the previous studies, current challenges, and provide insights for future works on the in-bed human pose estimation field.
arXiv Detail & Related papers (2024-02-01T15:57:11Z) - Explicit Occlusion Reasoning for Multi-person 3D Human Pose Estimation [33.86986028882488]
Occlusion poses a great threat to monocular multi-person 3D human pose estimation due to large variability in terms of the shape, appearance, and position of occluders.
Existing methods try to handle occlusion with pose priors/constraints, data augmentation, or implicit reasoning.
We develop a method to explicitly model this process that significantly improves bottom-up multi-person human pose estimation.
arXiv Detail & Related papers (2022-07-29T22:12:50Z) - Occluded Human Mesh Recovery [23.63235079216075]
We present Occluded Human Mesh Recovery (OCHMR) - a novel top-down mesh recovery approach that incorporates image spatial context.
OCHMR achieves superior performance on challenging multi-person benchmarks like 3DPW, CrowdPose and OCHuman.
arXiv Detail & Related papers (2022-03-24T21:39:20Z) - Privacy-Preserving In-Bed Pose Monitoring: A Fusion and Reconstruction
Study [9.474452908573111]
We explore the effective use of images from multiple non-visual and privacy-preserving modalities for the task of in-bed pose estimation.
First, we explore the effective fusion of information from different imaging modalities for better pose estimation.
Secondly, we propose a framework that can estimate in-bed pose estimation when visible images are unavailable.
arXiv Detail & Related papers (2022-02-22T07:24:21Z) - Towards Accurate Cross-Domain In-Bed Human Pose Estimation [3.685548851716087]
Long-wavelength infrared (LWIR) modality based pose estimation algorithms overcome the aforementioned challenges.
We propose a novel learning strategy comprises of two-fold data augmentation to reduce the cross-domain discrepancy.
Our experiments and analysis show the effectiveness of our approach over multiple standard human pose estimation baselines.
arXiv Detail & Related papers (2021-10-07T15:54:46Z) - AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in
the Wild [77.43884383743872]
We present AdaFuse, an adaptive multiview fusion method to enhance the features in occluded views.
We extensively evaluate the approach on three public datasets including Human3.6M, Total Capture and CMU Panoptic.
We also create a large scale synthetic dataset Occlusion-Person, which allows us to perform numerical evaluation on the occluded joints.
arXiv Detail & Related papers (2020-10-26T03:19:46Z) - Multi-person 3D Pose Estimation in Crowded Scenes Based on Multi-View
Geometry [62.29762409558553]
Epipolar constraints are at the core of feature matching and depth estimation in multi-person 3D human pose estimation methods.
Despite the satisfactory performance of this formulation in sparser crowd scenes, its effectiveness is frequently challenged under denser crowd circumstances.
In this paper, we depart from the multi-person 3D pose estimation formulation, and instead reformulate it as crowd pose estimation.
arXiv Detail & Related papers (2020-07-21T17:59:36Z) - HEMlets PoSh: Learning Part-Centric Heatmap Triplets for 3D Human Pose
and Shape Estimation [60.35776484235304]
This work attempts to address the uncertainty of lifting the detected 2D joints to the 3D space by introducing an intermediate state-Part-Centric Heatmap Triplets (HEMlets)
The HEMlets utilize three joint-heatmaps to represent the relative depth information of the end-joints for each skeletal body part.
A Convolutional Network (ConvNet) is first trained to predict HEMlets from the input image, followed by a volumetric joint-heatmap regression.
arXiv Detail & Related papers (2020-03-10T04:03:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.