Unsupervised 3D Pose Transfer with Cross Consistency and Dual Reconstruction

• URL: http://arxiv.org/abs/2211.10278v1
• Date: Fri, 18 Nov 2022 15:09:56 GMT
• Title: Unsupervised 3D Pose Transfer with Cross Consistency and Dual Reconstruction
• Authors: Chaoyue Song, Jiacheng Wei, Ruibo Li, Fayao Liu and Guosheng Lin
• Abstract summary: The goal of 3D pose transfer is to transfer the pose from the source mesh to the target mesh while preserving the identity information. Deep learning-based methods improved the efficiency and performance of 3D pose transfer. We present X-DualNet, a simple yet effective approach that enables unsupervised 3D pose transfer.
• Score: 50.94171353583328
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The goal of 3D pose transfer is to transfer the pose from the source mesh to the target mesh while preserving the identity information (e.g., face, body shape) of the target mesh. Deep learning-based methods improved the efficiency and performance of 3D pose transfer. However, most of them are trained under the supervision of the ground truth, whose availability is limited in real-world scenarios. In this work, we present X-DualNet, a simple yet effective approach that enables unsupervised 3D pose transfer. In X-DualNet, we introduce a generator $G$ which contains correspondence learning and pose transfer modules to achieve 3D pose transfer. We learn the shape correspondence by solving an optimal transport problem without any key point annotations and generate high-quality meshes with our elastic instance normalization (ElaIN) in the pose transfer module. With $G$ as the basic component, we propose a cross consistency learning scheme and a dual reconstruction objective to learn the pose transfer without supervision. Besides that, we also adopt an as-rigid-as-possible deformer in the training process to fine-tune the body shape of the generated results. Extensive experiments on human and animal data demonstrate that our framework can successfully achieve comparable performance as the state-of-the-art supervised approaches.

Related papers

• Enhancing Generalizability of Representation Learning for Data-Efficient 3D Scene Understanding [50.448520056844885]
  We propose a generative Bayesian network to produce diverse synthetic scenes with real-world patterns. A series of experiments robustly display our method's consistent superiority over existing state-of-the-art pre-training approaches.
  arXiv  Detail & Related papers  (2024-06-17T07:43:53Z)
• Towards Robust 3D Pose Transfer with Adversarial Learning [36.351835328908116]
  3D pose transfer that aims to transfer the desired pose to a target mesh is one of the most challenging 3D generation tasks. Previous attempts rely on well-defined parametric human models or skeletal joints as driving pose sources. We propose 3D-PoseMAE, a customized MAE that effectively learns 3D extrinsic presentations (i.e., pose)
  arXiv  Detail & Related papers  (2024-04-02T19:03:39Z)
• Cross-domain and Cross-dimension Learning for Image-to-Graph Transformers [50.576354045312115]
  Direct image-to-graph transformation is a challenging task that solves object detection and relationship prediction in a single model. We introduce a set of methods enabling cross-domain and cross-dimension transfer learning for image-to-graph transformers. We demonstrate our method's utility in cross-domain and cross-dimension experiments, where we pretrain our models on 2D satellite images before applying them to vastly different target domains in 2D and 3D.
  arXiv  Detail & Related papers  (2024-03-11T10:48:56Z)
• Weakly-supervised 3D Pose Transfer with Keypoints [57.66991032263699]
  Main challenges of 3D pose transfer are: 1) Lack of paired training data with different characters performing the same pose; 2) Disentangling pose and shape information from the target mesh; 3) Difficulty in applying to meshes with different topologies. We propose a novel weakly-supervised keypoint-based framework to overcome these difficulties.
  arXiv  Detail & Related papers  (2023-07-25T12:40:24Z)
• MAPConNet: Self-supervised 3D Pose Transfer with Mesh and Point Contrastive Learning [32.97354536302333]
  3D pose transfer is a challenging generation task that aims to transfer the pose of a source geometry onto a target geometry with the target identity preserved. Current pose transfer methods allow end-to-end correspondence learning but require the desired final output as ground truth for supervision. We present a novel self-supervised framework for 3D pose transfer which can be trained in unsupervised, semi-supervised, or fully supervised settings.
  arXiv  Detail & Related papers  (2023-04-26T20:42:40Z)
• PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervision [102.48681650013698]
  Existing self-supervised 3D human pose estimation schemes have largely relied on weak supervisions to guide the learning. We propose a novel self-supervised approach that allows us to explicitly generate 2D-3D pose pairs for augmenting supervision. This is made possible via introducing a reinforcement-learning-based imitator, which is learned jointly with a pose estimator alongside a pose hallucinator.
  arXiv  Detail & Related papers  (2022-03-29T14:45:53Z)
• Geometry-Contrastive Transformer for Generalized 3D Pose Transfer [95.56457218144983]
  The intuition of this work is to perceive the geometric inconsistency between the given meshes with the powerful self-attention mechanism. We propose a novel geometry-contrastive Transformer that has an efficient 3D structured perceiving ability to the global geometric inconsistencies. We present a latent isometric regularization module together with a novel semi-synthesized dataset for the cross-dataset 3D pose transfer task.
  arXiv  Detail & Related papers  (2021-12-14T13:14:24Z)
• 3D Pose Transfer with Correspondence Learning and Mesh Refinement [41.92922228475176]
  3D pose transfer is one of the most challenging 3D generation tasks. We propose a correspondence-refinement network to help the 3D pose transfer for both human and animal meshes.
  arXiv  Detail & Related papers  (2021-09-30T11:49:03Z)
• A 3D Mesh-based Lifting-and-Projection Network for Human Pose Transfer [25.681557081096805]
  We propose a lifting-and-projection framework to perform pose transfer in the 3D mesh space. To leverage the human body shape prior, LPNet exploits the topological information of the body mesh. To preserve texture details, ADCNet is introduced to enhance the feature produced by LPNet with the source foreground image.
  arXiv  Detail & Related papers  (2021-09-24T03:03:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.