Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks

• URL: http://arxiv.org/abs/2511.02830v1
• Date: Tue, 04 Nov 2025 18:58:03 GMT
• Title: Densemarks: Learning Canonical Embeddings for Human Heads Images via Point Tracks
• Authors: Dmitrii Pozdeev, Alexey Artemov, Ananta R. Bhattarai, Artem Sevastopolsky,
• Abstract summary: For a 2D image of a human head, a Vision Transformer network predicts a 3D embedding for each pixel, which corresponds to a location in a 3D canonical unit cube.<n>We employ multi-task learning with face landmarks and segmentation constraints, as well as imposing spatial continuity of embeddings.<n>The representation can be used for finding common semantic parts, face/head tracking, and stereo reconstruction.
• Score: 4.562267702525219
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose DenseMarks - a new learned representation for human heads, enabling high-quality dense correspondences of human head images. For a 2D image of a human head, a Vision Transformer network predicts a 3D embedding for each pixel, which corresponds to a location in a 3D canonical unit cube. In order to train our network, we collect a dataset of pairwise point matches, estimated by a state-of-the-art point tracker over a collection of diverse in-the-wild talking heads videos, and guide the mapping via a contrastive loss, encouraging matched points to have close embeddings. We further employ multi-task learning with face landmarks and segmentation constraints, as well as imposing spatial continuity of embeddings through latent cube features, which results in an interpretable and queryable canonical space. The representation can be used for finding common semantic parts, face/head tracking, and stereo reconstruction. Due to the strong supervision, our method is robust to pose variations and covers the entire head, including hair. Additionally, the canonical space bottleneck makes sure the obtained representations are consistent across diverse poses and individuals. We demonstrate state-of-the-art results in geometry-aware point matching and monocular head tracking with 3D Morphable Models. The code and the model checkpoint will be made available to the public.

Related papers

• Unsupervised Learning of Category-Level 3D Pose from Object-Centric Videos [15.532504015622159]
  Category-level 3D pose estimation is a fundamentally important problem in computer vision and robotics. We tackle the problem of learning to estimate the category-level 3D pose only from casually taken object-centric videos.
  arXiv  Detail & Related papers  (2024-07-05T09:43:05Z)
• Graphics Capsule: Learning Hierarchical 3D Face Representations from 2D Images [82.5266467869448]
  We propose an Inverse Graphics Capsule Network (IGC-Net) to learn the hierarchical 3D face representations from large-scale unlabeled images. IGC-Net first decomposes the objects into a set of semantic-consistent part-level descriptions and then assembles them into object-level descriptions to build the hierarchy.
  arXiv  Detail & Related papers  (2023-03-20T06:32:55Z)
• Pixel2ISDF: Implicit Signed Distance Fields based Human Body Model from Multi-view and Multi-pose Images [67.45882013828256]
  We focus on reconstructing clothed humans in the canonical space given multiple views and poses of a human as the input. We learn latent codes on the posed mesh by leveraging multiple input images and then assign the latent codes to the mesh in the canonical space. Our work for reconstructing the human shape on canonical pose achieves 3rd performance on WCPA MVP-Human Body Challenge.
  arXiv  Detail & Related papers  (2022-12-06T05:30:49Z)
• Learning Neural Parametric Head Models [7.679586286000453]
  We propose a novel 3D morphable model for complete human heads based on hybrid neural fields. We capture a person's identity in a canonical space as a signed distance field (SDF), and model facial expressions with a neural deformation field. Our representation achieves high-fidelity local detail by introducing an ensemble of local fields centered around facial anchor points.
  arXiv  Detail & Related papers  (2022-12-06T05:24:42Z)
• Unsupervised 3D Keypoint Discovery with Multi-View Geometry [104.76006413355485]
  We propose an algorithm that learns to discover 3D keypoints on human bodies from multiple-view images without supervision or labels. Our approach discovers more interpretable and accurate 3D keypoints compared to other state-of-the-art unsupervised approaches.
  arXiv  Detail & Related papers  (2022-11-23T10:25:12Z)
• ConDor: Self-Supervised Canonicalization of 3D Pose for Partial Shapes [55.689763519293464]
  ConDor is a self-supervised method that learns to canonicalize the 3D orientation and position for full and partial 3D point clouds. During inference, our method takes an unseen full or partial 3D point cloud at an arbitrary pose and outputs an equivariant canonical pose.
  arXiv  Detail & Related papers  (2022-01-19T18:57:21Z)
• Weakly Supervised Learning of Multi-Object 3D Scene Decompositions Using Deep Shape Priors [69.02332607843569]
  PriSMONet is a novel approach for learning Multi-Object 3D scene decomposition and representations from single images. A recurrent encoder regresses a latent representation of 3D shape, pose and texture of each object from an input RGB image. We evaluate the accuracy of our model in inferring 3D scene layout, demonstrate its generative capabilities, assess its generalization to real images, and point out benefits of the learned representation.
  arXiv  Detail & Related papers  (2020-10-08T14:49:23Z)
• Chained Representation Cycling: Learning to Estimate 3D Human Pose and Shape by Cycling Between Representations [73.11883464562895]
  We propose a new architecture that facilitates unsupervised, or lightly supervised, learning. We demonstrate the method by learning 3D human pose and shape from un-paired and un-annotated images. While we present results for modeling humans, our formulation is general and can be applied to other vision problems.
  arXiv  Detail & Related papers  (2020-01-06T14:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.