Direct 3D-Aware Object Insertion via Decomposed Visual Proxies

• URL: http://arxiv.org/abs/2606.06601v1
• Date: Thu, 04 Jun 2026 18:00:34 GMT
• Title: Direct 3D-Aware Object Insertion via Decomposed Visual Proxies
• Abstract summary: DIRECT (Decomposed Injection for Reference Composition and Target-integration) is a novel framework that integrates interactive pose manipulation with high-fidelity 2D image synthesis.<n>Our method decomposes the insertion conditions into three complementary components: appearance guidance capturing visual details from the reference object, geometry guidance derived from the user-adjusted 3D proxy, and context guidance from the target background.<n>Experiments show that DIRECT outperforms previous methods in both geometric controllability and visual quality.
• Score: 101.76006356727015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Object insertion aims to seamlessly composite a reference object into a specified region of a background image. Recent diffusion-based methods achieve high visual quality but formulate insertion as a simple 2D inpainting task, providing no explicit control over the object's 3D pose and limiting their practical applicability. We propose DIRECT (Decomposed Injection for Reference Composition and Target-integration), a novel framework that integrates interactive pose manipulation with high-fidelity 2D image synthesis to enable pose-controllable object insertion. Our method decomposes the insertion conditions into three complementary components: appearance guidance capturing visual details from the reference object, geometry guidance derived from the user-adjusted 3D proxy, and context guidance from the target background. By injecting them through separate pathways, DIRECT avoids feature entanglement and simultaneously preserves reference appearance, follows the user-specified pose, and adapts the object to the target scene. We also introduce an automated data construction pipeline to improve the diversity and quality of training data. Experiments show that DIRECT outperforms previous methods in both geometric controllability and visual quality.

Related papers

• Compact Object-Level Representations with Open-Vocabulary Understanding for Indoor Visual Relocalization [93.79141221821858]
  OpenReLoc is a camera relocalization system designed to provide scene understanding and accurate pose estimation capabilities.<n>We propose a multi-modal mechanism to integrate open-vocabulary semantic knowledge for effective 2D-3D object matching.<n> Experimental results demonstrate that OpenReLoc achieves superior relocalization recall and accuracy across various datasets.
  arXiv  Detail & Related papers  (2026-06-23T16:27:04Z)
• Pose-ICL: 3D-Aware In-Context Learning for Pose-Controllable Subject Customization [9.596779036612409]
  Pose-ICL is a tuning-free framework that adapts to new subjects through multiple paired image-pose references.<n>Its core mechanism,Surface-Anchored Position Embedding (SAPE), equips the model with explicit 3D awareness by anchoring image tokens to the surface coordinates of a volumetric bounding box.<n>Extensive evaluations on both 3D assets and real-world subjects demonstrate that Pose-ICL significantly outperforms current methods in both pose accuracy and identity consistency.
  arXiv  Detail & Related papers  (2026-06-09T14:13:40Z)
• Interact3D: Compositional 3D Generation of Interactive Objects [31.12099147294145]
  We present a novel framework designed to generate physically plausible interacting 3D compositional objects.<n>Our approach first leverages advanced generative priors to curate high-quality individual assets.<n>To physically compose these assets, we then introduce a robust two-stage composition pipeline.
  arXiv  Detail & Related papers  (2026-03-17T03:21:06Z)
• POCI-Diff: Position Objects Consistently and Interactively with 3D-Layout Guided Diffusion [46.97254555348757]
  We propose a diffusion-based approach for Text-to-Image (T2I) generation with consistent and interactive 3D layout control and editing.<n>We introduce a framework for Positioning Objects Consistently and Interactively (POCI-Diff)<n>Our method enables explicit per-object semantic control by binding individual text descriptions to specific 3D bounding boxes.
  arXiv  Detail & Related papers  (2026-01-20T15:13:43Z)
• HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation [50.206100327643284]
  HiScene is a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation.<n>We generate 3D content that aligns with 2D representations while maintaining compositional structure.
  arXiv  Detail & Related papers  (2025-04-17T16:33:39Z)
• IAAO: Interactive Affordance Learning for Articulated Objects in 3D Environments [56.85804719947]
  We present IAAO, a framework that builds an explicit 3D model for intelligent agents to gain understanding of articulated objects in their environment through interaction.<n>We first build hierarchical features and label fields for each object state using 3D Gaussian Splatting (3DGS) by distilling mask features and view-consistent labels from multi-view images.<n>We then perform object- and part-level queries on the 3D Gaussian primitives to identify static and articulated elements, estimating global transformations and local articulation parameters along with affordances.
  arXiv  Detail & Related papers  (2025-04-09T12:36:48Z)
• ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors [73.26004792375556]
  This paper shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. We demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects.
  arXiv  Detail & Related papers  (2023-08-24T17:59:51Z)
• OA-DET3D: Embedding Object Awareness as a General Plug-in for Multi-Camera 3D Object Detection [77.43427778037203]
  We introduce OA-DET3D, a plug-in module that improves 3D object detection.<n> OA-DET3D boosts the representation of objects by leveraging object-centric depth information and foreground pseudo points.<n>We conduct extensive experiments on the nuScenes dataset and Argoverse 2 dataset to validate the merits of OA-DET3D.
  arXiv  Detail & Related papers  (2023-01-13T06:02:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.