REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning

• URL: http://arxiv.org/abs/2504.04956v2
• Date: Tue, 08 Apr 2025 03:01:03 GMT
• Title: REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning
• Authors: Jihyun Lee, Weipeng Xu, Alexander Richard, Shih-En Wei, Shunsuke Saito, Shaojie Bai, Te-Li Wang, Minhyuk Sung, Tae-Kyun Kim, Jason Saragih,
• Abstract summary: We present REWIND, a one-step diffusion model for real-time, high-fidelity human motion estimation from egocentric image inputs.<n>We introduce cascaded body-hand denoising diffusion, which effectively models the correlation between egocentric body and hand motions.<n>We also propose a novel identity conditioning method based on a small set of pose exemplars of the target identity, which further enhances motion estimation quality.
• Score: 95.07708090428814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present REWIND (Real-Time Egocentric Whole-Body Motion Diffusion), a one-step diffusion model for real-time, high-fidelity human motion estimation from egocentric image inputs. While an existing method for egocentric whole-body (i.e., body and hands) motion estimation is non-real-time and acausal due to diffusion-based iterative motion refinement to capture correlations between body and hand poses, REWIND operates in a fully causal and real-time manner. To enable real-time inference, we introduce (1) cascaded body-hand denoising diffusion, which effectively models the correlation between egocentric body and hand motions in a fast, feed-forward manner, and (2) diffusion distillation, which enables high-quality motion estimation with a single denoising step. Our denoising diffusion model is based on a modified Transformer architecture, designed to causally model output motions while enhancing generalizability to unseen motion lengths. Additionally, REWIND optionally supports identity-conditioned motion estimation when identity prior is available. To this end, we propose a novel identity conditioning method based on a small set of pose exemplars of the target identity, which further enhances motion estimation quality. Through extensive experiments, we demonstrate that REWIND significantly outperforms the existing baselines both with and without exemplar-based identity conditioning.

Related papers

• Taming Consistency Distillation for Accelerated Human Image Animation [47.63111489003292]
  DanceLCM achieves results comparable to state-of-the-art video diffusion models with a mere 2-4 inference steps. The code and models will be made publicly available.
  arXiv  Detail & Related papers  (2025-04-15T12:44:53Z)
• ReCoM: Realistic Co-Speech Motion Generation with Recurrent Embedded Transformer [58.49950218437718]
  We present ReCoM, an efficient framework for generating high-fidelity and generalizable human body motions synchronized with speech.<n>The core innovation lies in the Recurrent Embedded Transformer (RET), which integrates Dynamic Embedding Regularization (DER) into a Vision Transformer (ViT) core architecture.<n>To enhance model robustness, we incorporate the proposed DER strategy, which equips the model with dual capabilities of noise resistance and cross-domain generalization.
  arXiv  Detail & Related papers  (2025-03-27T16:39:40Z)
• ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer [95.80384464922147]
  ACDiT is a blockwise Conditional Diffusion Transformer.<n>It offers a flexible between token-wise autoregression and full-sequence diffusion.<n>We show that ACDiT performs best among all autoregressive baselines on image and video generation tasks.
  arXiv  Detail & Related papers  (2024-12-10T18:13:20Z)
• RecMoDiffuse: Recurrent Flow Diffusion for Human Motion Generation [5.535590461577558]
  RecMoDiffuse is a new recurrent diffusion formulation for temporal modelling. We demonstrate the effectiveness of RecMoDiffuse in the temporal modelling of human motion.
  arXiv  Detail & Related papers  (2024-06-11T11:25:37Z)
• AnaMoDiff: 2D Analogical Motion Diffusion via Disentangled Denoising [25.839194626743126]
  AnaMoDiff is a novel diffusion-based method for 2D motion analogies. Our goal is to accurately transfer motions from a 2D driving video onto a source character, with its identity, in terms of appearance and natural movement.
  arXiv  Detail & Related papers  (2024-02-05T22:10:54Z)
• EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generation [57.539634387672656]
  Current state-of-the-art generative diffusion models have produced impressive results but struggle to achieve fast generation without sacrificing quality. We introduce Efficient Motion Diffusion Model (EMDM) for fast and high-quality human motion generation.
  arXiv  Detail & Related papers  (2023-12-04T18:58:38Z)
• Priority-Centric Human Motion Generation in Discrete Latent Space [59.401128190423535]
  We introduce a Priority-Centric Motion Discrete Diffusion Model (M2DM) for text-to-motion generation. M2DM incorporates a global self-attention mechanism and a regularization term to counteract code collapse. We also present a motion discrete diffusion model that employs an innovative noise schedule, determined by the significance of each motion token.
  arXiv  Detail & Related papers  (2023-08-28T10:40:16Z)
• Motion-DVAE: Unsupervised learning for fast human motion denoising [18.432026846779372]
  We introduce Motion-DVAE, a motion prior to capture the short-term dependencies of human motion. Together with Motion-DVAE, we introduce an unsupervised learned denoising method unifying regression- and optimization-based approaches.
  arXiv  Detail & Related papers  (2023-06-09T12:18:48Z)
• BoDiffusion: Diffusing Sparse Observations for Full-Body Human Motion Synthesis [14.331548412833513]
  Mixed reality applications require tracking the user's full-body motion to enable an immersive experience. We propose BoDiffusion -- a generative diffusion model for motion synthesis to tackle this under-constrained reconstruction problem. We present a time and space conditioning scheme that allows BoDiffusion to leverage sparse tracking inputs while generating smooth and realistic full-body motion sequences.
  arXiv  Detail & Related papers  (2023-04-21T16:39:05Z)
• Transformer Inertial Poser: Attention-based Real-time Human Motion Reconstruction from Sparse IMUs [79.72586714047199]
  We propose an attention-based deep learning method to reconstruct full-body motion from six IMU sensors in real-time. Our method achieves new state-of-the-art results both quantitatively and qualitatively, while being simple to implement and smaller in size.
  arXiv  Detail & Related papers  (2022-03-29T16:24:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.