GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction

• URL: http://arxiv.org/abs/2604.19624v1
• Date: Tue, 21 Apr 2026 16:13:15 GMT
• Title: GRAFT: Geometric Refinement and Fitting Transformer for Human Scene Reconstruction
• Abstract summary: We present GRAFT (Geometric Refinement And Fitting Transformer), a learned HSI prior that predicts Interaction Gradients.<n>A lightweight transformer recurrently updates human meshes and re-probes the scene, ensuring the final pose aligns with both learned priors and observed geometry.<n>Experiments show GRAFT improves interaction quality by up to 113% over state-of-the-art feed-forward methods.
• Score: 32.688160754014284
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reconstructing physically plausible 3D human-scene interactions (HSI) from a single image currently presents a trade-off: optimization based methods offer accurate contact but are slow (~20s), while feed-forward approaches are fast yet lack explicit interaction reasoning, producing floating and interpenetration artifacts. Our key insight is that geometry-based human--scene fitting can be amortized into fast feed-forward inference. We present GRAFT (Geometric Refinement And Fitting Transformer), a learned HSI prior that predicts Interaction Gradients: corrective parameter updates that iteratively refine human meshes by reasoning about their 3D relationship to the surrounding scene. GRAFT encodes the interaction state into compact body-anchored tokens, each grounded in the scene geometry via Geometric Probes that capture spatial relationships with nearby surfaces. A lightweight transformer recurrently updates human meshes and re-probes the scene, ensuring the final pose aligns with both learned priors and observed geometry. GRAFT operates either as an end-to-end reconstructor using image features, or with geometry alone as a transferable plug-and-play HSI prior that improves feed-forward methods without retraining. Experiments show GRAFT improves interaction quality by up to 113% over state-of-the-art feed-forward methods and matches optimization-based interaction quality at ${\sim}50{\times}$ lower runtime, while generalizing seamlessly to in-the-wild multi-person scenes and being preferred in 64.8% of three-way user study. Project page: https://pradyumnaym.github.io/graft .

Related papers

• TV-SGS: Gaussian Splatting with Geometric Information Propagation via Tensor Voting under sparse views [5.274411980510585]
  We introduce a new class of 3D losses that do not rely on rendering and can be combined with essentially all losses previously reported in the literature.<n>Our method is easy to integrate with a diverse set of backbones, and our experiments on the DTU and Tanks-and-Temples datasets demonstrate that TV-SGS improves the geometry of the outputs compared to the backbone.
  arXiv  Detail & Related papers  (2026-09-07T16:40:51Z)
• Artic-O: End-to-End Articulated Object Reconstruction via Latent Geometry Learning [105.35529379360186]
  Artic-O is an end-to-end, feed-forward framework for articulated object reconstruction.<n>It maps sparse multi-state observations into a pretrained latent geometry space.<n>A frozen flow-matching decoder provides a complete-shape prior for recovering visible and occluded structures.
  arXiv  Detail & Related papers  (2026-06-20T08:15:30Z)
• Face Anything: 4D Face Reconstruction from Any Image Sequence [49.395407357499074]
  We present a unified method for high-fidelity 4D facial reconstruction based on canonical facial point prediction.<n>By jointly predicting depth and canonical coordinates, our method enables accurate depth estimation, temporally stable reconstruction, dense 3D geometry, and robust facial point tracking.
  arXiv  Detail & Related papers  (2026-04-21T17:22:39Z)
• GraphiContact: Pose-aware Human-Scene Robust Contact Perception for Interactive Systems [17.089112324421468]
  We propose a pose-aware framework that transfers human priors from two pretrained Transformer encoders and predicts per-vertex human-scene contact on the reconstructed mesh.<n> Experiments on five benchmark datasets show that GraphiContact achieves consistent gains on both contact prediction and 3D human reconstruction.<n>Our code, based on the GraphiContact method, provides comprehensive 3D human reconstruction and interaction analysis.
  arXiv  Detail & Related papers  (2026-03-19T17:17:04Z)
• CRISP: Contact-Guided Real2Sim from Monocular Video with Planar Scene Primitives [65.89192712575797]
  We introduce CRISP, a method that recovers simulatable human motion and scene geometry from monocular video.<n>Our approach reduces motion tracking failure rates from 55.2% to 6.9% on human-centric video benchmarks.<n>This demonstrates CRISP's ability to generate physically-valid human motion and interaction environments at scale.
  arXiv  Detail & Related papers  (2025-12-16T18:59:50Z)
• AlignGS: Aligning Geometry and Semantics for Robust Indoor Reconstruction from Sparse Views [18.361136390711415]
  The demand for semantically rich 3D models of indoor scenes is rapidly growing, driven by applications in augmented reality, virtual reality, and robotics.<n>Existing methods often treat semantics as a passive feature painted on an already-formed, and potentially flawed, geometry.<n>This paper introduces AlignGS, a novel framework that actualizes this vision by pioneering a synergistic, end-to-end optimization of geometry and semantics.
  arXiv  Detail & Related papers  (2025-10-09T06:30:20Z)
• HART: Human Aligned Reconstruction Transformer [17.065147884544853]
  HART is a unified framework for sparse-view human reconstruction.<n>It outputs a watertight clothed mesh, the aligned SMPL-X body mesh, and a Gaussian-splat representation for novel-view rendering.<n>Results suggest that feed-forward transformers can serve as a scalable model for robust human reconstruction in real-world settings.
  arXiv  Detail & Related papers  (2025-09-30T17:56:02Z)
• Aligned Novel View Image and Geometry Synthesis via Cross-modal Attention Instillation [62.87088388345378]
  We introduce a diffusion-based framework that performs aligned novel view image and geometry generation via a warping-and-inpainting methodology.<n>Method leverages off-the-shelf geometry predictors to predict partial geometries viewed from reference images.<n>Cross-modal attention distillation is proposed to ensure accurate alignment between generated images and geometry.
  arXiv  Detail & Related papers  (2025-06-13T16:19:00Z)
• GRACE: Estimating Geometry-level 3D Human-Scene Contact from 2D Images [54.602947113980655]
  Estimating the geometry level of human-scene contact aims to ground specific contact surface points at 3D human geometries.<n> GRACE (Geometry-level Reasoning for 3D Human-scene Contact Estimation) is a new paradigm for 3D human contact estimation.<n>It incorporates a point cloud encoder-decoder architecture along with a hierarchical feature extraction and fusion module.
  arXiv  Detail & Related papers  (2025-05-10T09:25:46Z)
• CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image [44.8172828045897]
  Current methods often struggle with domain-specific limitations or low-quality object generation.<n>We propose CAST, a novel method for 3D scene reconstruction and recovery.
  arXiv  Detail & Related papers  (2025-02-18T14:29:52Z)
• InstantSplat: Sparse-view Gaussian Splatting in Seconds [91.77050739918037]
  We introduce InstantSplat, a novel approach for addressing sparse-view 3D scene reconstruction at lightning-fast speed.<n>InstantSplat employs a self-supervised framework that optimize 3D scene representation and camera poses.<n>It achieves an acceleration of over 30x in reconstruction and improves visual quality (SSIM) from 0.3755 to 0.7624 compared to traditional SfM with 3D-GS.
  arXiv  Detail & Related papers  (2024-03-29T17:29:58Z)
• FrozenRecon: Pose-free 3D Scene Reconstruction with Frozen Depth Models [67.96827539201071]
  We propose a novel test-time optimization approach for 3D scene reconstruction. Our method achieves state-of-the-art cross-dataset reconstruction on five zero-shot testing datasets.
  arXiv  Detail & Related papers  (2023-08-10T17:55:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.