Imagining the Unseen: Generative Location Modeling for Object Placement

• URL: http://arxiv.org/abs/2410.13564v2
• Date: Tue, 07 Oct 2025 09:29:02 GMT
• Title: Imagining the Unseen: Generative Location Modeling for Object Placement
• Authors: Jooyeol Yun, Davide Abati, Mohamed Omran, Jaegul Choo, Amirhossein Habibian, Auke Wiggers,
• Abstract summary: We develop a generative location model that learns to predict plausible bounding boxes for an object.<n>Our approach first tokenizes the image and target object class, then decodes bounding box coordinates through an autoregressive transformer.<n> Empirical evaluations reveal that our generative location model achieves superior placement accuracy on the OPA dataset.
• Score: 49.71690795831461
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Location modeling, or determining where non-existing objects could feasibly appear in a scene, has the potential to benefit numerous computer vision tasks, from automatic object insertion to scene creation in virtual reality. Yet, this capability remains largely unexplored to date. In this paper, we develop a generative location model that, given an object class and an image, learns to predict plausible bounding boxes for such an object. Our approach first tokenizes the image and target object class, then decodes bounding box coordinates through an autoregressive transformer. This formulation effectively addresses two core challenges in locatio modeling: the inherent one-to-many nature of plausible locations, and the sparsity of existing location modeling datasets, where fewer than 1% of valid placements are labeled. Furthermore, we incorporate Direct Preference Optimization to leverage negative labels, refining the spatial predictions. Empirical evaluations reveal that our generative location model achieves superior placement accuracy on the OPA dataset as compared to discriminative baselines and image composition approaches. We further test our model in the context of object insertion, where it proposes locations for an off-the-shelf inpainting model to render objects. In this respect, our proposal exhibits improved visual coherence relative to state-of-the-art instruction-tuned editing methods, demonstrating a high-performing location model's utility in a downstream application.

Related papers

• Controllable 3D Placement of Objects with Scene-Aware Diffusion Models [6.020146107338903]
  We show that a carefully designed visual map, combined with coarse object masks, is sufficient for high quality object placement.<n>We show that fine location control can be combined with appearance control to place existing objects in precise locations in a scene.
  arXiv  Detail & Related papers  (2025-06-26T16:31:39Z)
• BOOTPLACE: Bootstrapped Object Placement with Detection Transformers [23.300369070771836]
  We introduce BOOTPLACE, a novel paradigm that formulates object placement as a placement-by-detection problem.<n> Experimental results on established benchmarks demonstrate BOOTPLACE's superior performance in object repositioning.
  arXiv  Detail & Related papers  (2025-03-27T21:21:20Z)
• ObjectMover: Generative Object Movement with Video Prior [69.75281888309017]
  We present ObjectMover, a generative model that can perform object movement in challenging scenes. We show that with this approach, our model is able to adjust to complex real-world scenarios. We propose a multi-task learning strategy that enables training on real-world video data to improve the model generalization.
  arXiv  Detail & Related papers  (2025-03-11T04:42:59Z)
• AnyPlace: Learning Generalized Object Placement for Robot Manipulation [29.482987292744568]
  We propose AnyPlace, a two-stage method trained entirely on synthetic data.<n>Our key insight is that by leveraging a Vision-Language Model, we focus only on the relevant regions for local placement.<n>For training, we generate a fully synthetic dataset of randomly generated objects in different placement configurations.<n>In real-world experiments, we show how our approach directly transfers models trained purely on synthetic data to the real world.
  arXiv  Detail & Related papers  (2025-02-06T22:04:13Z)
• Add-it: Training-Free Object Insertion in Images With Pretrained Diffusion Models [78.90023746996302]
  Add-it is a training-free approach that extends diffusion models' attention mechanisms to incorporate information from three key sources. Our weighted extended-attention mechanism maintains structural consistency and fine details while ensuring natural object placement. Human evaluations show that Add-it is preferred in over 80% of cases.
  arXiv  Detail & Related papers  (2024-11-11T18:50:09Z)
• EraseDraw: Learning to Insert Objects by Erasing Them from Images [24.55843674256795]
  Prior works often fail by making global changes to the image, inserting objects in unrealistic spatial locations, and generating inaccurate lighting details. We observe that while state-of-the-art models perform poorly on object insertion, they can remove objects and erase the background in natural images very well. We show compelling results on diverse insertion prompts and images across various domains.
  arXiv  Detail & Related papers  (2024-08-31T18:37:48Z)
• DiffUHaul: A Training-Free Method for Object Dragging in Images [78.93531472479202]
  We propose a training-free method, dubbed DiffUHaul, for the object dragging task. We first apply attention masking in each denoising step to make the generation more disentangled across different objects. In the early denoising steps, we interpolate the attention features between source and target images to smoothly fuse new layouts with the original appearance.
  arXiv  Detail & Related papers  (2024-06-03T17:59:53Z)
• Few-shot Object Localization [37.347898735345574]
  This paper defines a novel task named Few-Shot Object localization (FSOL) It aims to achieve precise localization with limited samples. This task achieves generalized object localization by leveraging a small number of labeled support samples to query the positional information of objects within corresponding images. Experimental results demonstrate a significant performance improvement of our approach in the FSOL task, establishing an efficient benchmark for further research.
  arXiv  Detail & Related papers  (2024-03-19T05:50:48Z)
• FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects [55.77542145604758]
  FoundationPose is a unified foundation model for 6D object pose estimation and tracking. Our approach can be instantly applied at test-time to a novel object without fine-tuning.
  arXiv  Detail & Related papers  (2023-12-13T18:28:09Z)
• Weakly-supervised Contrastive Learning for Unsupervised Object Discovery [52.696041556640516]
  Unsupervised object discovery is promising due to its ability to discover objects in a generic manner. We design a semantic-guided self-supervised learning model to extract high-level semantic features from images. We introduce Principal Component Analysis (PCA) to localize object regions.
  arXiv  Detail & Related papers  (2023-07-07T04:03:48Z)
• TopNet: Transformer-based Object Placement Network for Image Compositing [43.14411954867784]
  Local clues in background images are important to determine the compatibility of placing objects with certain locations/scales. We propose to learn the correlation between object features and all local background features with a transformer module. Our new formulation generates a 3D heatmap indicating the plausibility of all location/scale combinations in one network forward pass.
  arXiv  Detail & Related papers  (2023-04-06T20:58:49Z)
• MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare [84.80956484848505]
  MegaPose is a method to estimate the 6D pose of novel objects, that is, objects unseen during training. We present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.
  arXiv  Detail & Related papers  (2022-12-13T19:30:03Z)
• ObjectStitch: Generative Object Compositing [43.206123360578665]
  We propose a self-supervised framework for object compositing using conditional diffusion models. Our framework can transform the viewpoint, geometry, color and shadow of the generated object while requiring no manual labeling. Our method outperforms relevant baselines in both realism and faithfulness of the synthesized result images in a user study on various real-world images.
  arXiv  Detail & Related papers  (2022-12-02T02:15:13Z)
• Towards Self-Supervised Category-Level Object Pose and Size Estimation [121.28537953301951]
  This work presents a self-supervised framework for category-level object pose and size estimation from a single depth image. We leverage the geometric consistency residing in point clouds of the same shape for self-supervision.
  arXiv  Detail & Related papers  (2022-03-06T06:02:30Z)
• Learning Models as Functionals of Signed-Distance Fields for Manipulation Planning [51.74463056899926]
  This work proposes an optimization-based manipulation planning framework where the objectives are learned functionals of signed-distance fields that represent objects in the scene. We show that representing objects as signed-distance fields not only enables to learn and represent a variety of models with higher accuracy compared to point-cloud and occupancy measure representations.
  arXiv  Detail & Related papers  (2021-10-02T12:36:58Z)
• Localizing Infinity-shaped fishes: Sketch-guided object localization in the wild [5.964436882344729]
  This work investigates the problem of sketch-guided object localization. Human sketches are used as queries to conduct the object localization in natural images. We propose a sketch-conditioned DETR architecture which avoids a hard classification. We experimentally demonstrate that our model and its variants significantly advance over previous state-of-the-art results.
  arXiv  Detail & Related papers  (2021-09-24T10:39:43Z)
• Salient Objects in Clutter [130.63976772770368]
  This paper identifies and addresses a serious design bias of existing salient object detection (SOD) datasets. This design bias has led to a saturation in performance for state-of-the-art SOD models when evaluated on existing datasets. We propose a new high-quality dataset and update the previous saliency benchmark.
  arXiv  Detail & Related papers  (2021-05-07T03:49:26Z)
• Object-Centric Image Generation from Layouts [93.10217725729468]
  We develop a layout-to-image-generation method to generate complex scenes with multiple objects. Our method learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We introduce SceneFID, an object-centric adaptation of the popular Fr'echet Inception Distance metric, that is better suited for multi-object images.
  arXiv  Detail & Related papers  (2020-03-16T21:40:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.