THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond

• URL: http://arxiv.org/abs/2603.25892v1
• Date: Thu, 26 Mar 2026 20:28:40 GMT
• Title: THFM: A Unified Video Foundation Model for 4D Human Perception and Beyond
• Abstract summary: We present THFM, a unified video foundation model for human-centric perception.<n>Our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks.<n>For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals.
• Score: 22.66024130777808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present THFM, a unified video foundation model for human-centric perception that jointly addresses dense tasks (depth, normals, segmentation, dense pose) and sparse tasks (2d/3d keypoint estimation) within a single architecture. THFM is derived from a pretrained text-to-video diffusion model, repurposed as a single-forward-pass perception model and augmented with learnable tokens for sparse predictions. Modulated by the text prompt, our single unified model is capable of performing various perception tasks. Crucially, our model is on-par or surpassing state-of-the-art specialized models on a variety of benchmarks despite being trained exclusively on synthetic data (i.e.~without training on real-world or benchmark specific data). We further highlight intriguing emergent properties of our model, which we attribute to the underlying diffusion-based video representation. For example, our model trained on videos with a single human in the scene generalizes to multiple humans and other object classes such as anthropomorphic characters and animals -- a capability that hasn't been demonstrated in the past.

Related papers

• Video Generation Models are General-Purpose Vision Learners [62.28137589984863]
  We argue that large-scale text-to-video generation serves as a strong pre-training paradigm for computer vision.<n>We introduce GenCeption, which leverages a pre-trained video video diffusion backbone to define a feed-forward perception model.<n> Empirical results demonstrate that GenCeption achieves state-of-the-art performance across a diverse suite of tasks.
  arXiv  Detail & Related papers  (2026-07-10T01:09:06Z)
• Instruct-Particulate: Scaling Feed-Forward 3D Object Articulation with Kinematic Control [86.64250947574749]
  Reconstructing articulated 3D objects is important for animation, gaming, and robotic simulations.<n>Recent neural networks can estimate the articulated structure of 3D objects, but their generalization remains limited by the scarcity of annotated data.<n>We introduce Instruct-Particulate, a model that takes a 3D mesh together with a target kinematic specification.
  arXiv  Detail & Related papers  (2026-06-12T17:59:36Z)
• Attend to Anything: Foundation Model for Unified Human Attention Modeling [23.199307554176016]
  We present the Attend to Anything Model (AAM), a multi-modal foundation model that unifies attention modeling across various image, video, and audio-visual tasks and scenes.<n>AAM reformulates attention as a cognitive entailment relationship organized in a general-to-specific hierarchy, implemented through language prompts with hierarchical embeddings in hyperbolic space.<n>Experiments on 16 benchmarks demonstrate that AAM consistently outperforms state-of-the-art methods by an average of 6% across various scenarios.
  arXiv  Detail & Related papers  (2026-06-02T12:00:21Z)
• Human-Object Interaction Detection Collaborated with Large Relation-driven Diffusion Models [65.82564074712836]
  We introduce DIFfusionHOI, a new HOI detector shedding light on text-to-image diffusion models. We first devise an inversion-based strategy to learn the expression of relation patterns between humans and objects in embedding space. These learned relation embeddings then serve as textual prompts, to steer diffusion models generate images that depict specific interactions.
  arXiv  Detail & Related papers  (2024-10-26T12:00:33Z)
• VLOGGER: Multimodal Diffusion for Embodied Avatar Synthesis [40.869862603815875]
  VLOGGER is a method for audio-driven human video generation from a single input image. We use a novel diffusion-based architecture that augments text-to-image models with both spatial and temporal controls. We show applications in video editing and personalization.
  arXiv  Detail & Related papers  (2024-03-13T17:59:02Z)
• NViST: In the Wild New View Synthesis from a Single Image with Transformers [8.361847255300846]
  We propose NViST, a transformer-based model for efficient novel-view synthesis from a single image. NViST is trained on MVImgNet, a large-scale dataset of casually-captured real-world videos. We show results on unseen objects and categories from MVImgNet and even generalization to casual phone captures.
  arXiv  Detail & Related papers  (2023-12-13T23:41:17Z)
• Cross-view and Cross-pose Completion for 3D Human Understanding [22.787947086152315]
  We propose a pre-training approach based on self-supervised learning that works on human-centric data using only images. We pre-train a model for body-centric tasks and one for hand-centric tasks. With a generic transformer architecture, these models outperform existing self-supervised pre-training methods on a wide set of human-centric downstream tasks.
  arXiv  Detail & Related papers  (2023-11-15T16:51:18Z)
• Human Performance Capture from Monocular Video in the Wild [50.34917313325813]
  We propose a method capable of capturing the dynamic 3D human shape from a monocular video featuring challenging body poses. Our method outperforms state-of-the-art methods on an in-the-wild human video dataset 3DPW.
  arXiv  Detail & Related papers  (2021-11-29T16:32:41Z)
• Unsupervised Object Learning via Common Fate [61.14802390241075]
  Learning generative object models from unlabelled videos is a long standing problem and required for causal scene modeling. We decompose this problem into three easier subtasks, and provide candidate solutions for each of them. We show that our approach allows learning generative models that generalize beyond the occlusions present in the input videos.
  arXiv  Detail & Related papers  (2021-10-13T08:22:04Z)
• imGHUM: Implicit Generative Models of 3D Human Shape and Articulated Pose [42.4185273307021]
  We present imGHUM, the first holistic generative model of 3D human shape and articulated pose. We model the full human body implicitly as a function zero-level-set and without the use of an explicit template mesh.
  arXiv  Detail & Related papers  (2021-08-24T17:08:28Z)
• Learning Local Recurrent Models for Human Mesh Recovery [50.85467243778406]
  We present a new method for video mesh recovery that divides the human mesh into several local parts following the standard skeletal model. We then model the dynamics of each local part with separate recurrent models, with each model conditioned appropriately based on the known kinematic structure of the human body. This results in a structure-informed local recurrent learning architecture that can be trained in an end-to-end fashion with available annotations.
  arXiv  Detail & Related papers  (2021-07-27T14:30:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.