Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video

• URL: http://arxiv.org/abs/2603.17178v1
• Date: Tue, 17 Mar 2026 22:10:55 GMT
• Title: Patient4D: Temporally Consistent Patient Body Mesh Recovery from Monocular Operating Room Video
• Abstract summary: Patient4D is a station-constrained reconstruction pipeline that exploits the stationarity prior.<n>Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline.<n>Our findings demonstrate that exploiting stationarity priors can substantially improve monocular reconstruction in clinical AR scenarios.
• Score: 9.229366844390803
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recovering a dense 3D body mesh from monocular video remains challenging under occlusion from draping and continuously moving camera viewpoints. This configuration arises in surgical augmented reality (AR), where an anesthetized patient lies under surgical draping while a surgeon's head-mounted camera continuously changes viewpoint. Existing human mesh recovery (HMR) methods are typically trained on upright, moving subjects captured from relatively stable cameras, leading to performance degradation under such conditions. To address this, we present Patient4D, a stationarity-constrained reconstruction pipeline that explicitly exploits the stationarity prior. The pipeline combines image-level foundation models for perception with lightweight geometric mechanisms that enforce temporal consistency across frames. Two key components enable robust reconstruction: Pose Locking, which anchors pose parameters using stable keyframes, and Rigid Fallback, which recovers meshes under severe occlusion through silhouette-guided rigid alignment. Together, these mechanisms stabilize predictions while remaining compatible with off-the-shelf HMR models. We evaluate Patient4D on 4,680 synthetic surgical sequences and three public HMR video benchmarks. Under surgical drape occlusion, Patient4D achieves a 0.75 mean IoU, reducing failure frames from 30.5% to 1.3% compared to the best baseline. Our findings demonstrate that exploiting stationarity priors can substantially improve monocular reconstruction in clinical AR scenarios.

Related papers

• From Pixels to Newtons: Predicting In Vivo Joint Contact Forces from Monocular Video [0.6768558752130311]
  I present a physics-free pipeline to predict instantaneous 3D hip and knee contact forces from an uncalibrated monocular video.<n>Parametric body meshes are recovered per frame, encoded as kinematic features, and decoded into forces by a transformer.<n>This pipeline establishes uncalibrated monocular video as a viable modality for estimating joint loading.
  arXiv  Detail & Related papers  (2026-06-04T18:31:00Z)
• FactorizedHMR: A Hybrid Framework for Video Human Mesh Recovery [6.1625435030904745]
  Human Mesh Recovery (HMR) is fundamentally ambiguous.<n>We propose FactorizedHMR, a two-stage framework that treats these two regimes differently.<n>We show that FactorizedHMR remains competitive with strong baselines across camera-space and world-space benchmarks.
  arXiv  Detail & Related papers  (2026-05-14T13:59:56Z)
• ResiHMR: Residual-Limb Aware Single-Image 3D Human Mesh Recovery for Individuals with Limb Loss [28.590347498910006]
  We present ResiHMR, a residual-limb aware framework for single-image 3D human modeling.<n>To the best of our knowledge, this is the first single-image HMR system that explicitly reconstructs residual-limb surfaces.
  arXiv  Detail & Related papers  (2026-04-30T15:42:17Z)
• SurgCUT3R: Surgical Scene-Aware Continuous Understanding of Temporal 3D Representation [20.17317715120012]
  SurgCUT3R is a systematic framework that adapts unified 3D reconstruction models to the surgical domain.<n>First, we develop a data generation pipeline that exploits public stereo surgical datasets.<n>Second, we propose a hybrid supervision strategy that couples our pseudo-ground-truth with geometric self-correction.<n>Third, we introduce a hierarchical inference framework that employs two specialized models to effectively accumulated pose drift over long surgical videos.
  arXiv  Detail & Related papers  (2026-03-07T01:13:18Z)
• ArtHOI: Articulated Human-Object Interaction Synthesis by 4D Reconstruction from Video Priors [51.06020148149403]
  We introduce ArtHOI, the first zero-shot framework for articulated human-object interaction synthesis via 4D reconstruction from video priors.<n>ArtHOI bridges video-based generation and geometry-aware reconstruction, producing interactions that are both semantically aligned and physically grounded.
  arXiv  Detail & Related papers  (2026-03-04T17:58:04Z)
• Diff2DGS: Reliable Reconstruction of Occluded Surgical Scenes via 2D Gaussian Splatting [10.70948053935438]
  We propose Diff2DGS, a novel two-stage framework for reliable 3D reconstruction of occluded surgical scenes.<n>In the first stage, a diffusion-based video module with temporal priors inpaints tissue occluded by instruments with high spatial-temporal consistency.<n>In the second stage, we adapt 2D Gaussian Splatting (2DGS) with a Learnable Deformation Model (LDM) to capture dynamic tissue deformation and anatomical geometry.<n>D Diff2DGS outperforms state-of-the-art approaches in both appearance and geometry, reaching 38.02 dB PSNR on
  arXiv  Detail & Related papers  (2026-02-20T16:14:21Z)
• Towards Geometry-Aware and Motion-Guided Video Human Mesh Recovery [60.51998732898099]
  We introduce HMRMamba, a new paradigm for 3D Human Mesh Recovery.<n>It pioneers the use of Structured State Space Models for their efficiency and long-range modeling prowess.<n>Our framework is distinguished by two core contributions. First, the Geometry-Aware Lifting Module, featuring a novel dual-scan Mamba architecture.
  arXiv  Detail & Related papers  (2026-01-29T08:05:02Z)
• SAM-Body4D: Training-Free 4D Human Body Mesh Recovery from Videos [53.227781131348856]
  Human Mesh Recovery aims to reconstruct 3D human pose and shape from 2D observations.<n>Recent image-based HMR methods such as SAM 3D Body achieve strong robustness on in-the-wild images.<n>We propose SAM-Body4D, a training-free framework for temporally consistent and occlusion-robust HMR from videos.
  arXiv  Detail & Related papers  (2025-12-09T09:37:31Z)
• CPT-4DMR: Continuous sPatial-Temporal Representation for 4D-MRI Reconstruction [11.238319012891324]
  We introduce a neural framework that considers respiratory motion as a smooth, continuous deformation steered by a 1D surrogate signal.<n>The proposed method significantly improves efficiency, reducing the total processing time from approximately five hours required to just 15 minutes of training.<n>It achieves superior performance compared to conventional methods, and demonstrates strong potential for application in 4D radiation therapy planning and real-time adaptive treatment.
  arXiv  Detail & Related papers  (2025-09-22T21:18:26Z)
• CryoFastAR: Fast Cryo-EM Ab Initio Reconstruction Made Easy [58.86576215228903]
  We introduce CryoFastAR, the first geometric foundation model that can directly predict poses from Cryo-EM noisy images for Fast ab initio Reconstruction.<n>By integrating multi-view features and training on large-scale simulated cryo-EM data with realistic noise and CTF modulations, CryoFastAR enhances pose estimation accuracy and generalization.
  arXiv  Detail & Related papers  (2025-06-06T08:32:32Z)
• Endo3R: Unified Online Reconstruction from Dynamic Monocular Endoscopic Video [35.241054116681426]
  Endo3R is a unified 3D foundation model for online scale-consistent reconstruction from monocular surgical video.<n>Our model unifies the tasks by predicting globally aligned pointmaps, scale-consistent video depths, and camera parameters without any offline optimization.
  arXiv  Detail & Related papers  (2025-04-04T06:05:22Z)
• 3D Vessel Reconstruction from Sparse-View Dynamic DSA Images via Vessel Probability Guided Attenuation Learning [79.60829508459753]
  Current commercial Digital Subtraction Angiography (DSA) systems typically demand hundreds of scanning views to perform reconstruction. The dynamic blood flow and insufficient input of sparse-view DSA images present significant challenges to the 3D vessel reconstruction task. We propose to use a time-agnostic vessel probability field to solve this problem effectively.
  arXiv  Detail & Related papers  (2024-05-17T11:23:33Z)
• FLex: Joint Pose and Dynamic Radiance Fields Optimization for Stereo Endoscopic Videos [79.50191812646125]
  Reconstruction of endoscopic scenes is an important asset for various medical applications, from post-surgery analysis to educational training. We adress the challenging setup of a moving endoscope within a highly dynamic environment of deforming tissue. We propose an implicit scene separation into multiple overlapping 4D neural radiance fields (NeRFs) and a progressive optimization scheme jointly optimizing for reconstruction and camera poses from scratch. This improves the ease-of-use and allows to scale reconstruction capabilities in time to process surgical videos of 5,000 frames and more; an improvement of more than ten times compared to the state of the art while being agnostic to external tracking information
  arXiv  Detail & Related papers  (2024-03-18T19:13:02Z)
• A Quantitative Evaluation of Dense 3D Reconstruction of Sinus Anatomy from Monocular Endoscopic Video [8.32570164101507]
  We perform a quantitative analysis of a self-supervised approach for sinus reconstruction using endoscopic sequences and optical tracking. Our results show that the generated reconstructions are in high agreement with the anatomy, yielding an average point-to-mesh error of 0.91 mm. We identify that pose and depth estimation inaccuracies contribute equally to this error and that locally consistent sequences with shorter trajectories generate more accurate reconstructions.
  arXiv  Detail & Related papers  (2023-10-22T17:11:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.