MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modeling

• URL: http://arxiv.org/abs/2406.16137v1
• Date: Sun, 23 Jun 2024 15:18:30 GMT
• Title: MLPHand: Real Time Multi-View 3D Hand Mesh Reconstruction via MLP Modeling
• Abstract summary: Multi-view hand reconstruction is a critical task for applications in virtual reality human-computer interaction. HandHand consists of two primary modules: (1) a lightweight-based Skeleton2Mesh model that efficiently recovers hand meshes from hand skeletons, and (2) a multi-view geometry feature fusion prediction module. Experiments on three widely used datasets demonstrate that HandHand can reduce reconstruction by 90% while achieving comparable accuracy to existing state-of-the-art baselines complexity.
• Score: 54.54190158459534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-view hand mesh reconstruction is a critical task for applications in virtual reality and human-computer interaction, but it remains a formidable challenge. Although existing multi-view hand reconstruction methods achieve remarkable accuracy, they typically come with an intensive computational burden that hinders real-time inference. To this end, we propose MLPHand, a novel method designed for real-time multi-view single hand reconstruction. MLP Hand consists of two primary modules: (1) a lightweight MLP-based Skeleton2Mesh model that efficiently recovers hand meshes from hand skeletons, and (2) a multi-view geometry feature fusion prediction module that enhances the Skeleton2Mesh model with detailed geometric information from multiple views. Experiments on three widely used datasets demonstrate that MLPHand can reduce computational complexity by 90% while achieving comparable reconstruction accuracy to existing state-of-the-art baselines.

Related papers

• Reliev3R: Relieving Feed-forward Reconstruction from Multi-View Geometric Annotations [98.66466590444553]
  We propose Reliev3R, a weakly-supervised paradigm for training FFRMs from scratch without cost-prohibitive multi-view geometric annotations.<n>At the core of Reliev3R, we design an ambiguity-aware relative depth loss and a trigonometry-based reprojection loss to facilitate supervision for multi-view geometric consistency.
  arXiv  Detail & Related papers  (2026-04-01T06:46:54Z)
• HandMCM: Multi-modal Point Cloud-based Correspondence State Space Model for 3D Hand Pose Estimation [60.2305990057581]
  3D hand pose estimation is crucial for many human-computer interaction applications such as augmented reality.<n>HandMCM is a novel method based on the powerful state space model (Mamba)
  arXiv  Detail & Related papers  (2026-02-02T03:25:43Z)
• Towards Geometry-Aware and Motion-Guided Video Human Mesh Recovery [60.51998732898099]
  We introduce HMRMamba, a new paradigm for 3D Human Mesh Recovery.<n>It pioneers the use of Structured State Space Models for their efficiency and long-range modeling prowess.<n>Our framework is distinguished by two core contributions. First, the Geometry-Aware Lifting Module, featuring a novel dual-scan Mamba architecture.
  arXiv  Detail & Related papers  (2026-01-29T08:05:02Z)
• EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild [79.71523320368388]
  Our work aims to reconstruct hand-object interactions from a single-view image. We first design a novel pipeline to estimate the underlying hand pose and object shape. With the initial reconstruction, we employ a prior-guided optimization scheme.
  arXiv  Detail & Related papers  (2024-11-21T16:33:35Z)
• 3D Points Splatting for Real-Time Dynamic Hand Reconstruction [13.392046706568275]
  3D Points Splatting Hand Reconstruction (3D-PSHR) is a real-time and photo-realistic hand reconstruction approach. We propose a self-adaptive canonical points up strategy to achieve high-resolution hand geometry representation. To model texture, we disentangle the appearance color into the intrinsic albedo and pose-aware shading.
  arXiv  Detail & Related papers  (2023-12-21T11:50:49Z)
• HandMIM: Pose-Aware Self-Supervised Learning for 3D Hand Mesh Estimation [5.888156950854715]
  We propose a novel self-supervised pre-training strategy for regressing 3D hand mesh parameters. Our proposed approach, named HandMIM, achieves strong performance on various hand mesh estimation tasks.
  arXiv  Detail & Related papers  (2023-07-29T19:46:06Z)
• Overcoming the Trade-off Between Accuracy and Plausibility in 3D Hand Shape Reconstruction [62.96478903239799]
  Direct mesh fitting for 3D hand shape reconstruction is highly accurate. However, the reconstructed meshes are prone to artifacts and do not appear as plausible hand shapes. We introduce a novel weakly-supervised hand shape estimation framework that integrates non-parametric mesh fitting with MANO model in an end-to-end fashion.
  arXiv  Detail & Related papers  (2023-05-01T03:38:01Z)
• R2-MLP: Round-Roll MLP for Multi-View 3D Object Recognition [33.53114929452528]
  Vision architectures based exclusively on multi-layer perceptrons (MLPs) have gained much attention in the computer vision community. We present an achieves a view-based 3D object recognition task by considering the communications between patches from different views. With a conceptually simple structure, our R$2$MLP achieves competitive performance compared with existing methods.
  arXiv  Detail & Related papers  (2022-11-20T21:13:02Z)
• End-to-end Weakly-supervised Single-stage Multiple 3D Hand Mesh Reconstruction from a Single RGB Image [9.238322841389994]
  We propose a single-stage pipeline for multi-hand reconstruction. Specifically, we design a multi-head auto-encoder structure, where each head network shares the same feature map and outputs the hand center, pose and texture. Our method outperforms the state-of-the-art model-based methods in both weakly-supervised and fully-supervised manners.
  arXiv  Detail & Related papers  (2022-04-18T03:57:14Z)
• Direct Multi-view Multi-person 3D Pose Estimation [138.48139701871213]
  We present Multi-view Pose transformer (MvP) for estimating multi-person 3D poses from multi-view images. MvP directly regresses the multi-person 3D poses in a clean and efficient way, without relying on intermediate tasks. We show experimentally that our MvP model outperforms the state-of-the-art methods on several benchmarks while being much more efficient.
  arXiv  Detail & Related papers  (2021-11-07T13:09:20Z)
• Towards Accurate Alignment in Real-time 3D Hand-Mesh Reconstruction [57.3636347704271]
  3D hand-mesh reconstruction from RGB images facilitates many applications, including augmented reality (AR) This paper presents a novel pipeline by decoupling the hand-mesh reconstruction task into three stages. We can promote high-quality finger-level mesh-image alignment and drive the models together to deliver real-time predictions.
  arXiv  Detail & Related papers  (2021-09-03T20:42:01Z)
• Joint Hand-object 3D Reconstruction from a Single Image with Cross-branch Feature Fusion [78.98074380040838]
  We propose to consider hand and object jointly in feature space and explore the reciprocity of the two branches. We employ an auxiliary depth estimation module to augment the input RGB image with the estimated depth map. Our approach significantly outperforms existing approaches in terms of the reconstruction accuracy of objects.
  arXiv  Detail & Related papers  (2020-06-28T09:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.