HandFlow: Fully Generative 4D Hand Recovery with Flow Matching

• URL: http://arxiv.org/abs/2607.11221v1
• Date: Mon, 13 Jul 2026 08:13:42 GMT
• Title: HandFlow: Fully Generative 4D Hand Recovery with Flow Matching
• Abstract summary: HandFlow is a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video.<n>It achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness.<n>On a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method.
• Score: 25.56265773611543
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Accurate monocular 4D hand reconstruction remains challenging. Per-frame discriminative regressors lack temporal context and often produce jittery predictions. Temporal models improve consistency by aggregating information across frames, but they are typically deterministic regressors, making them vulnerable to ambiguous observations caused by occlusion and motion blur. Generative modeling offers a natural alternative by learning a prior over plausible hand motion sequences, enabling coherent hand-state recovery when visual evidence is incomplete or unreliable. Motivated by this observation, we present HandFlow, a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video. Given visual and skeletal observations, HandFlow denoises an entire temporal window of MANO parameters through a single ODE integration. To support this, we use a Flux-style dual-stream transformer that attends across the full sequence to capture long-range dependencies without autoregressive decoding, and a confidence-aware continuous masking mechanism that blends observed features with learnable mask tokens to handle noisy or missing observations. Experiments on DexYCB and HOT3D show that HandFlow achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness. It reduces world-space pose error by over 30% compared with the strongest baseline and achieves the lowest acceleration error among all evaluated methods, while remaining competitive in per-frame pose accuracy. Moreover, on a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method, with reconstruction itself accounting for only a small fraction of the end-to-end latency.

Related papers

• MotionMAR: Multi-scale Auto-Regressive Human Motion Reconstruction from Sparse Observations [51.837372131319285]
  MotionMAR is a framework for motion reconstruction from sparse observations.<n>It first estimates the global trajectory of human motion and then gradually refines the temporal details.<n>MotionMAR achieves state-of-the-art accuracy on the AMASS dataset.
  arXiv  Detail & Related papers  (2026-06-22T08:15:56Z)
• Feed-forward Motion In-betweening for Any 4D [34.801496888575166]
  We propose an in-between framework for arbitrary 4D meshes with conditioning.<n>We show strong performance and improved controllability on DyMesh16 and DyMesh32 benchmarks.
  arXiv  Detail & Related papers  (2026-06-20T16:18:13Z)
• RAFT-MSF++: Temporal Geometry-Motion Feature Fusion for Self-Supervised Monocular Scene Flow [51.43025173196566]
  Monocular scene flow estimation aims to recover dense 3D motion from image sequences.<n>We propose RAFT-MSF++, a self-supervised multi-frame framework that fuses temporal features to jointly estimate depth and scene flow.<n>Experiments show that RAFT-MSF++ achieves 24.14% SF-all on the KITTI Scene Flow benchmark, with a 30.99% improvement over the baseline and better robustness in occluded regions.
  arXiv  Detail & Related papers  (2026-04-21T11:32:49Z)
• Grasp in Gaussians: Fast Monocular Reconstruction of Dynamic Hand-Object Interactions [66.64605515970584]
  We present Grasp in Gaussians, a robust method for reconstructing dynamic 3D hand-object interactions from a single monocular video.<n>Our key insight is that accurate and temporally stable hand-object motion can be recovered using a compact Sum-of-Gaussians representation.<n>Experiments on public benchmarks demonstrate that GraG reconstructs temporally coherent hand-object interactions on long sequences 6.4x faster than prior work.
  arXiv  Detail & Related papers  (2026-04-14T16:19:12Z)
• End-to-End Spatial-Temporal Transformer for Real-time 4D HOI Reconstruction [74.31251139839047]
  THO is an end-to-end Spatial-Temporal Transformer that predicts human motion and coordinated object motion in a forward fashion from the given video and 3D template.<n> experiments demonstrate that THO operates at an inference speed of 31.5 FPS on a single 4090 GPU, achieving a >600x speedup over prior optimization-based methods.
  arXiv  Detail & Related papers  (2026-03-15T15:21:36Z)
• Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos [27.595035122927204]
  We present a feed-forward 4D human reconstruction and model that efficientlycalibrates temporally aligned representations from uncalibrated sparse-view videos.<n>For novel time, we design a novel motion prediction module to predict dense motions for each 3D Gaussian between two adjacent frames.<n>Experiments demonstrate the effectiveness of our model on both in-domain and out-of-domain datasets.
  arXiv  Detail & Related papers  (2025-09-29T02:47:14Z)
• SuperFlow++: Enhanced Spatiotemporal Consistency for Cross-Modal Data Pretraining [62.433137130087445]
  SuperFlow++ is a novel framework that integrates pretraining and downstream tasks using consecutive camera pairs.<n>We show that SuperFlow++ outperforms state-of-the-art methods across diverse tasks and driving conditions.<n>With strong generalizability and computational efficiency, SuperFlow++ establishes a new benchmark for data-efficient LiDAR-based perception in autonomous driving.
  arXiv  Detail & Related papers  (2025-03-25T17:59:57Z)
• Enhanced Spatio-Temporal Context for Temporally Consistent Robust 3D Human Motion Recovery from Monocular Videos [5.258814754543826]
  We propose a novel method for temporally consistent motion estimation from a monocular video. Instead of using generic ResNet-like features, our method uses a body-aware feature representation and an independent per-frame pose. Our method attains significantly lower acceleration error and outperforms the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-11-20T10:53:59Z)
• OcclusionFusion: Occlusion-aware Motion Estimation for Real-time Dynamic 3D Reconstruction [14.130915525776055]
  RGBD-based real-time dynamic 3D reconstruction suffers from inaccurate inter-frame motion estimation. We propose OcclusionFusion, a novel method to calculate occlusion-aware 3D motion to guide the reconstruction. Our technique outperforms existing single-view-based real-time methods by a large margin.
  arXiv  Detail & Related papers  (2022-03-15T15:09:01Z)
• Consistency Guided Scene Flow Estimation [159.24395181068218]
  CGSF is a self-supervised framework for the joint reconstruction of 3D scene structure and motion from stereo video. We show that the proposed model can reliably predict disparity and scene flow in challenging imagery. It achieves better generalization than the state-of-the-art, and adapts quickly and robustly to unseen domains.
  arXiv  Detail & Related papers  (2020-06-19T17:28:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.