On the Use of Hierarchical Vision Foundation Models for Low-Cost Human Mesh Recovery and Pose Estimation

• URL: http://arxiv.org/abs/2510.12660v1
• Date: Tue, 14 Oct 2025 15:57:40 GMT
• Title: On the Use of Hierarchical Vision Foundation Models for Low-Cost Human Mesh Recovery and Pose Estimation
• Abstract summary: We aim to develop simple and efficient models for human mesh recovery (HMR) and human pose estimation (HPE)<n>We first construct three lightweight HMR2.0 variants by adapting the corresponding ViTPose models.<n>In addition, we propose leveraging the early stages of hierarchical vision foundation models (VFMs) as encoders.
• Score: 1.6385815610837169
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we aim to develop simple and efficient models for human mesh recovery (HMR) and its predecessor task, human pose estimation (HPE). State-of-the-art HMR methods, such as HMR2.0 and its successors, rely on large, non-hierarchical vision transformers as encoders, which are inherited from the corresponding HPE models like ViTPose. To establish baselines across varying computational budgets, we first construct three lightweight HMR2.0 variants by adapting the corresponding ViTPose models. In addition, we propose leveraging the early stages of hierarchical vision foundation models (VFMs), including Swin Transformer, GroupMixFormer, and VMamba, as encoders. This design is motivated by the observation that intermediate stages of hierarchical VFMs produce feature maps with resolutions comparable to or higher than those of non-hierarchical counterparts. We conduct a comprehensive evaluation of 27 hierarchical-VFM-based HMR and HPE models, demonstrating that using only the first two or three stages achieves performance on par with full-stage models. Moreover, we show that the resulting truncated models exhibit better trade-offs between accuracy and computational efficiency compared to existing lightweight alternatives.

Related papers

• Marigold V2: Revisiting Diffusion Transformers for Monocular Depth Estimation [68.73567966621631]
  We revisit Marigold, a set of techniques for repurposing modern image generation and editing models, into state-of-the-art monocular depth estimators.<n>Our recipes target single-step inference from pretrained multi-step flow-matching models, with quantization where needed, preserving model capacity while remaining cheap to run.<n>The results are crisper, cleaner depth maps that generalize well out-of-distribution, with 16-26% improvement in AbsRel over the previous best on KITTI and ETH3D.
  arXiv  Detail & Related papers  (2026-09-08T00:52:01Z)
• HY-Embodied-0.5: Embodied Foundation Models for Real-World Agents [71.09493646932046]
  We introduce HY-Embodied-0.5, a family of foundation models specifically designed for real-world embodied agents.<n>The HY-Embodied-0.5 suite comprises two primary variants: an efficient model with 2B activated parameters designed for edge deployment, and a powerful model with 32B activated parameters targeted for complex reasoning.<n>Our MoT-2B model outperforms similarly sized state-of-the-art models on 16 benchmarks, while the 32B variant achieves performance comparable to frontier models such as Gemini 3.0 Pro.
  arXiv  Detail & Related papers  (2026-04-08T17:59:48Z)
• Modeling Cross-vision Synergy for Unified Large Vision Model [130.37489011094036]
  PolyV is a unified large vision model that achieves cross-vision synergy at both the architectural and training levels.<n>PolyV consistently outperforms existing models, achieving over 10% average improvement over its backbone.
  arXiv  Detail & Related papers  (2026-03-03T22:44:43Z)
• UniSH: Unifying Scene and Human Reconstruction in a Feed-Forward Pass [83.7071371474926]
  UniSH is a unified, feed-forward framework for joint metric-scale 3D scene and human reconstruction.<n>Our framework bridges strong, disparate priors from scene reconstruction and HMR.<n>Our model achieves state-of-the-art performance on human-centric scene reconstruction.
  arXiv  Detail & Related papers  (2026-01-03T16:06:27Z)
• ReasonX: MLLM-Guided Intrinsic Image Decomposition [41.961378846441015]
  Intrinsic image decomposition aims to separate images into physical components such as albedo, depth, normals, and illumination.<n>We propose ReasonX, a novel framework that leverages a large language model (MLLM) as a perceptual judge providing relative intrinsic comparisons.<n>Our framework aligns conditional intrinsic predictors by rewarding agreement between the judge's assessments and analytically derived relations from the model's outputs.
  arXiv  Detail & Related papers  (2025-12-03T19:44:07Z)
• Vision Foundation Models Can Be Good Tokenizers for Latent Diffusion Models [45.63522160275318]
  Vision Foundation Model Variational Autoencoder (VFM-VAE) designed to resolve inherent tension between VFM's semantic focus and need for pixel-level fidelity.<n>Our system reaches a gFID (w/o CFG) of 2.20 in merely 80 epochs (a 10x speedup over prior tokenizers)
  arXiv  Detail & Related papers  (2025-10-21T09:30:45Z)
• Dynamic Pattern Alignment Learning for Pretraining Lightweight Human-Centric Vision Models [84.30626369903221]
  We propose Dynamic Pattern Alignment Learning (DPAL) to efficiently train lightweight human-centric vision models.<n>The DPAL guides lightweight HVMs to learn all typical human visual patterns from large HVMs, which can generalize to various human-centric vision tasks.<n>Extensive experiments conducted on 15 challenging datasets demonstrate the effectiveness of the DPAL.
  arXiv  Detail & Related papers  (2025-08-10T02:27:06Z)
• Improving Progressive Generation with Decomposable Flow Matching [50.63174319509629]
  Decomposable Flow Matching (DFM) is a simple and effective framework for the progressive generation of visual media.<n>On Imagenet-1k 512px, DFM achieves 35.2% improvements in FDD scores over the base architecture and 26.4% over the best-performing baseline.
  arXiv  Detail & Related papers  (2025-06-24T17:58:02Z)
• UAVTwin: Neural Digital Twins for UAVs using Gaussian Splatting [57.63613048492219]
  We present UAVTwin, a method for creating digital twins from real-world environments and facilitating data augmentation for training downstream models embedded in unmanned aerial vehicles (UAVs)<n>This is achieved by integrating 3D Gaussian Splatting (3DGS) for reconstructing backgrounds along with controllable synthetic human models that display diverse appearances and actions in multiple poses.
  arXiv  Detail & Related papers  (2025-04-02T22:17:30Z)
• Flow Generator Matching [35.371071097381346]
  Flow Generator Matching (FGM) is designed to accelerate the sampling of flow-matching models into a one-step generation. On the CIFAR10 unconditional generation benchmark, our one-step FGM model achieves a new record Fr'echet Inception Distance (FID) score of 3.08. MM-DiT-FGM one-step text-to-image model demonstrates outstanding industry-level performance.
  arXiv  Detail & Related papers  (2024-10-25T05:41:28Z)
• SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgery [54.866490321241905]
  Model merging-based multitask learning (MTL) offers a promising approach for performing MTL by merging multiple expert models. In this paper, we examine the merged model's representation distribution and uncover a critical issue of "representation bias" This bias arises from a significant distribution gap between the representations of the merged and expert models, leading to the suboptimal performance of the merged MTL model.
  arXiv  Detail & Related papers  (2024-10-18T11:49:40Z)
• SMPLer: Taming Transformers for Monocular 3D Human Shape and Pose Estimation [74.07836010698801]
  We propose an SMPL-based Transformer framework (SMPLer) to address this issue. SMPLer incorporates two key ingredients: a decoupled attention operation and an SMPL-based target representation. Extensive experiments demonstrate the effectiveness of SMPLer against existing 3D human shape and pose estimation methods.
  arXiv  Detail & Related papers  (2024-04-23T17:59:59Z)
• Multiscale Metamorphic VAE for 3D Brain MRI Synthesis [5.060516201839319]
  Generative modeling of 3D brain MRIs presents difficulties in achieving high visual fidelity while ensuring sufficient coverage of the data distribution. In this work, we propose to address this challenge with composable, multiscale morphological transformations in a variational autoencoder framework. We show substantial performance improvements in FID while retaining comparable, or superior, reconstruction quality compared to prior work based on VAEs and generative adversarial networks (GANs)
  arXiv  Detail & Related papers  (2023-01-09T09:15:30Z)
• Distribution-Aware Single-Stage Models for Multi-Person 3D Pose Estimation [29.430404703883084]
  We present a novel Distribution-Aware Single-stage (DAS) model for tackling the challenging multi-person 3D pose estimation problem. The proposed DAS model simultaneously localizes person positions and their corresponding body joints in the 3D camera space in a one-pass manner. Comprehensive experiments on benchmarks CMU Panoptic and MuPoTS-3D demonstrate the superior efficiency of the proposed DAS model.
  arXiv  Detail & Related papers  (2022-03-15T07:30:27Z)
• THUNDR: Transformer-based 3D HUmaN Reconstruction with Markers [67.8628917474705]
  THUNDR is a transformer-based deep neural network methodology to reconstruct the 3d pose and shape of people. We show state-of-the-art results on Human3.6M and 3DPW, for both the fully-supervised and the self-supervised models. We observe very solid 3d reconstruction performance for difficult human poses collected in the wild.
  arXiv  Detail & Related papers  (2021-06-17T09:09:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.