PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing

• URL: http://arxiv.org/abs/2604.07230v2
• Date: Thu, 09 Apr 2026 11:01:59 GMT
• Title: PhyEdit: Towards Real-World Object Manipulation via Physically-Grounded Image Editing
• Abstract summary: We develop PhyEdit, an image editing framework that leverages explicit geometric simulation as contextual 3D-aware visual guidance.<n>By combining this plug-and-play 3D prior with joint 2D--3D supervision, our method effectively improves physical accuracy and manipulation consistency.<n>We present a real-world dataset, RealManip-10K, for 3D-aware object manipulation featuring paired images and depth annotations.
• Score: 24.0524480042673
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Achieving physically accurate object manipulation in image editing is essential for its potential applications in interactive world models. However, existing visual generative models often fail at precise spatial manipulation, resulting in incorrect scaling and positioning of objects. This limitation primarily stems from the lack of explicit mechanisms to incorporate 3D geometry and perspective projection. To achieve accurate manipulation, we develop PhyEdit, an image editing framework that leverages explicit geometric simulation as contextual 3D-aware visual guidance. By combining this plug-and-play 3D prior with joint 2D--3D supervision, our method effectively improves physical accuracy and manipulation consistency. To support this method and evaluate performance, we present a real-world dataset, RealManip-10K, for 3D-aware object manipulation featuring paired images and depth annotations. We also propose ManipEval, a benchmark with multi-dimensional metrics to evaluate 3D spatial control and geometric consistency. Extensive experiments show that our approach outperforms existing methods, including strong closed-source models, in both 3D geometric accuracy and manipulation consistency.

Related papers

• SpatialDiff: 3D-Aware Object Movement via Implicit Spatial Modeling [60.4038189368113]
  We introduce Implicit 3D Spatial Modeling, which introduces 3D prior knowledge and enables the model to internally build a comprehensive understanding of the three-dimensional spatial structure.<n>We also present Global Spatial Supervision, which constrains the latent spatial features to enable the model to perceive changes in object spatial positions caused by editing operations.<n> Experimental results demonstrate that our method significantly improves the accuracy and fidelity of spatial movement in complex scenes.
  arXiv  Detail & Related papers  (2026-08-22T04:46:08Z)
• Towards Realistic and Consistent Orbital Video Generation via 3D Foundation Priors [61.34273238077091]
  We present a novel method for generating orbital videos from a single image of an object.<n>Our method achieves superior visual quality, shape realism and multi-view consistency compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-04-14T05:35:46Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• Ctrl&Shift: High-Quality Geometry-Aware Object Manipulation in Visual Generation [34.92056161129864]
  We present Ctrl&Shift, an end-to-end diffusion framework to achieve geometry-consistent object manipulation without explicit 3D representations.<n>Our key insight is to decompose manipulation into two stages, object removal and reference-guided inpainting under explicit camera pose control, and encode both within a unified diffusion process.<n>To our knowledge, this is the first framework to unify fine-grained geometric control and real-world generalization for object manipulation, without relying on any explicit 3D modeling.
  arXiv  Detail & Related papers  (2026-02-11T23:36:30Z)
• Free-Form Scene Editor: Enabling Multi-Round Object Manipulation like in a 3D Engine [83.0145525456509]
  We present FFSE, a 3D-aware framework designed to enable intuitive, physically-consistent object editing on real-world images.<n>Unlike previous approaches that either operate in image space or require slow and error-prone 3D reconstruction, FFSE models editing as a sequence of learned 3D transformations.<n>To support learning of multi-round 3D-aware object manipulation, we introduce 3DObjectEditor.
  arXiv  Detail & Related papers  (2025-11-17T18:57:39Z)
• 3DGeoDet: General-purpose Geometry-aware Image-based 3D Object Detection [17.502554516157893]
  3DGeoDet is a novel geometry-aware 3D object detection approach.<n>It effectively handles single- and multi-view RGB images in indoor and outdoor environments.
  arXiv  Detail & Related papers  (2025-06-11T09:18:36Z)
• Enhancing Single Image to 3D Generation using Gaussian Splatting and Hybrid Diffusion Priors [17.544733016978928]
  3D object generation from a single image involves estimating the full 3D geometry and texture of unseen views from an unposed RGB image captured in the wild. Recent advancements in 3D object generation have introduced techniques that reconstruct an object's 3D shape and texture. We propose bridging the gap between 2D and 3D diffusion models to address this limitation.
  arXiv  Detail & Related papers  (2024-10-12T10:14:11Z)
• DO3D: Self-supervised Learning of Decomposed Object-aware 3D Motion and Depth from Monocular Videos [76.01906393673897]
  We propose a self-supervised method to jointly learn 3D motion and depth from monocular videos. Our system contains a depth estimation module to predict depth, and a new decomposed object-wise 3D motion (DO3D) estimation module to predict ego-motion and 3D object motion. Our model delivers superior performance in all evaluated settings.
  arXiv  Detail & Related papers  (2024-03-09T12:22:46Z)
• RiCS: A 2D Self-Occlusion Map for Harmonizing Volumetric Objects [68.85305626324694]
  Ray-marching in Camera Space (RiCS) is a new method to represent the self-occlusions of foreground objects in 3D into a 2D self-occlusion map. We show that our representation map not only allows us to enhance the image quality but also to model temporally coherent complex shadow effects.
  arXiv  Detail & Related papers  (2022-05-14T05:35:35Z)
• AutoShape: Real-Time Shape-Aware Monocular 3D Object Detection [15.244852122106634]
  We propose an approach for incorporating the shape-aware 2D/3D constraints into the 3D detection framework. Specifically, we employ the deep neural network to learn distinguished 2D keypoints in the 2D image domain. For generating the ground truth of 2D/3D keypoints, an automatic model-fitting approach has been proposed.
  arXiv  Detail & Related papers  (2021-08-25T08:50:06Z)
• Learning Geometry-Guided Depth via Projective Modeling for Monocular 3D Object Detection [70.71934539556916]
  We learn geometry-guided depth estimation with projective modeling to advance monocular 3D object detection. Specifically, a principled geometry formula with projective modeling of 2D and 3D depth predictions in the monocular 3D object detection network is devised. Our method remarkably improves the detection performance of the state-of-the-art monocular-based method without extra data by 2.80% on the moderate test setting.
  arXiv  Detail & Related papers  (2021-07-29T12:30:39Z)
• Geometric Processing for Image-based 3D Object Modeling [2.6397379133308214]
  This article focuses on introducing the state-of-the-art methods of three major components of geometric processing: 1) geo-referencing; 2) Image dense matching 3) texture mapping. The largely automated geometric processing of images in a 3D object reconstruction workflow, is becoming a critical part of the reality-based 3D modeling.
  arXiv  Detail & Related papers  (2021-06-27T18:33:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.