Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion

• URL: http://arxiv.org/abs/2412.14462v1
• Date: Thu, 19 Dec 2024 02:23:13 GMT
• Title: Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion
• Abstract summary: We extend the concept of Affordance from human-centered image composition tasks to a more general object-scene composition framework.<n>We propose a Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream architecture to simultaneously denoise the RGB image and the insertion mask.<n>Our method outperforms the state-of-the-art methods and exhibits strong generalization performance on in-the-wild images.
• Score: 29.770096013143117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a common image editing operation, image composition involves integrating foreground objects into background scenes. In this paper, we expand the application of the concept of Affordance from human-centered image composition tasks to a more general object-scene composition framework, addressing the complex interplay between foreground objects and background scenes. Following the principle of Affordance, we define the affordance-aware object insertion task, which aims to seamlessly insert any object into any scene with various position prompts. To address the limited data issue and incorporate this task, we constructed the SAM-FB dataset, which contains over 3 million examples across more than 3,000 object categories. Furthermore, we propose the Mask-Aware Dual Diffusion (MADD) model, which utilizes a dual-stream architecture to simultaneously denoise the RGB image and the insertion mask. By explicitly modeling the insertion mask in the diffusion process, MADD effectively facilitates the notion of affordance. Extensive experimental results show that our method outperforms the state-of-the-art methods and exhibits strong generalization performance on in-the-wild images. Please refer to our code on https://github.com/KaKituken/affordance-aware-any.

Related papers

• Direct 3D-Aware Object Insertion via Decomposed Visual Proxies [101.76006356727015]
  DIRECT (Decomposed Injection for Reference Composition and Target-integration) is a novel framework that integrates interactive pose manipulation with high-fidelity 2D image synthesis.<n>Our method decomposes the insertion conditions into three complementary components: appearance guidance capturing visual details from the reference object, geometry guidance derived from the user-adjusted 3D proxy, and context guidance from the target background.<n>Experiments show that DIRECT outperforms previous methods in both geometric controllability and visual quality.
  arXiv  Detail & Related papers  (2026-06-04T18:00:34Z)
• ObjEmbed: Towards Universal Multimodal Object Embeddings [74.39703419628829]
  We present Embed, a novel individual object embedding model.<n>It decomposes the input image into multiple regional embeddings, each corresponding to an individual object, along with global embeddings.<n>It supports a wide range of visual understanding tasks like visual retrieval, local image retrieval, and global image retrieval.
  arXiv  Detail & Related papers  (2026-02-02T07:38:45Z)
• PLACID: Identity-Preserving Multi-Object Compositing via Video Diffusion with Synthetic Trajectories [22.63777279327245]
  PLACID is a framework that transforms a collection of object images into an appealing multi-object composite.<n>First, we leverage a pretrained image-to-video (I2V) diffusion model with text control to preserve objects consistency, identities, and background details.<n>Second, we propose a novel data curation strategy that generates synthetic sequences where randomly placed objects smoothly move to their target positions.
  arXiv  Detail & Related papers  (2026-01-30T19:42:54Z)
• SGDFuse: SAM-Guided Diffusion for High-Fidelity Infrared and Visible Image Fusion [65.80051636480836]
  This paper proposes a conditional diffusion model guided by the Segment Anything Model (SAM) to achieve high-fidelity and semantically-aware image fusion.<n>The framework operates in a two-stage process: it first performs a preliminary fusion of multi-modal features, and then utilizes the semantic masks as a condition to drive the diffusion model's coarse-to-fine denoising generation.<n>Extensive experiments demonstrate that SGDFuse achieves state-of-the-art performance in both subjective and objective evaluations.
  arXiv  Detail & Related papers  (2025-08-07T10:58:52Z)
• ObjectClear: Complete Object Removal via Object-Effect Attention [56.2893552300215]
  We introduce a new dataset for OBject-Effect Removal, named OBER, which provides paired images with and without object effects, along with precise masks for both objects and their associated visual artifacts.<n>We propose a novel framework, ObjectClear, which incorporates an object-effect attention mechanism to guide the model toward the foreground removal regions by learning attention masks.<n>Experiments demonstrate that ObjectClear outperforms existing methods, achieving improved object-effect removal quality and background fidelity, especially in complex scenarios.
  arXiv  Detail & Related papers  (2025-05-28T17:51:17Z)
• Insert Anything: Image Insertion via In-Context Editing in DiT [19.733787045511775]
  We present a unified framework for reference-based image insertion that seamlessly integrates objects from reference images into target scenes under flexible, user-specified control guidance. Our approach is trained once on our new AnyInsertion dataset--comprising 120K prompt-image pairs covering diverse tasks such as person, object, and garment insertion--and effortlessly generalizes to a wide range of insertion scenarios.
  arXiv  Detail & Related papers  (2025-04-21T10:19:12Z)
• HiScene: Creating Hierarchical 3D Scenes with Isometric View Generation [50.206100327643284]
  HiScene is a novel hierarchical framework that bridges the gap between 2D image generation and 3D object generation. We generate 3D content that aligns with 2D representations while maintaining compositional structure.
  arXiv  Detail & Related papers  (2025-04-17T16:33:39Z)
• A Diffusion-Based Framework for Occluded Object Movement [39.6345172890042]
  We propose a Diffusion-based framework specifically designed for Occluded Object Movement, named DiffOOM. The de-occlusion branch utilizes a background color-fill strategy and a continuously updated object mask to focus the diffusion process on completing the obscured portion of the target object. Concurrently, the movement branch employs latent optimization to place the completed object in the target location and adopts local text-conditioned guidance to integrate the object into new surroundings appropriately.
  arXiv  Detail & Related papers  (2025-04-02T16:29:30Z)
• ObjectMate: A Recurrence Prior for Object Insertion and Subject-Driven Generation [33.91045409317844]
  This paper introduces a tuning-free method for both object insertion and subject-driven generation.<n>The task involves composing an object, given multiple views, into a scene specified by either an image or text.<n>We compare our method with state-of-the-art methods for object insertion and subject-driven generation, using a single or multiple references.
  arXiv  Detail & Related papers  (2024-12-11T18:59:53Z)
• Improving Text-guided Object Inpainting with Semantic Pre-inpainting [95.17396565347936]
  We decompose the typical single-stage object inpainting into two cascaded processes: semantic pre-inpainting and high-fieldity object generation. To achieve this, we cascade a Transformer-based semantic inpainter and an object inpainting diffusion model, leading to a novel CAscaded Transformer-Diffusion framework.
  arXiv  Detail & Related papers  (2024-09-12T17:55:37Z)
• Pluralistic Salient Object Detection [108.74650817891984]
  We introduce pluralistic salient object detection (PSOD), a novel task aimed at generating multiple plausible salient segmentation results for a given input image. We present two new SOD datasets "DUTS-MM" and "DUS-MQ", along with newly designed evaluation metrics.
  arXiv  Detail & Related papers  (2024-09-04T01:38:37Z)
• Completing Visual Objects via Bridging Generation and Segmentation [84.4552458720467]
  MaskComp delineates the completion process through iterative stages of generation and segmentation. In each iteration, the object mask is provided as an additional condition to boost image generation. We demonstrate that the combination of one generation and one segmentation stage effectively functions as a mask denoiser.
  arXiv  Detail & Related papers  (2023-10-01T22:25:40Z)
• DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing [94.24479528298252]
  DragGAN is an interactive point-based image editing framework that achieves impressive editing results with pixel-level precision. By harnessing large-scale pretrained diffusion models, we greatly enhance the applicability of interactive point-based editing on both real and diffusion-generated images. We present a challenging benchmark dataset called DragBench to evaluate the performance of interactive point-based image editing methods.
  arXiv  Detail & Related papers  (2023-06-26T06:04:09Z)
• AMICO: Amodal Instance Composition [40.03865667370814]
  Image composition aims to blend multiple objects to form a harmonized image. We present Amodal Instance Composition for blending imperfect objects onto a target image. Our results show state-of-the-art performance on public COCOA and KINS benchmarks.
  arXiv  Detail & Related papers  (2022-10-11T23:23:14Z)
• Exploring the Interactive Guidance for Unified and Effective Image Matting [16.933897631478146]
  We propose a Unified Interactive image Matting method, named UIM, which solves the limitations and achieves satisfying matting results. Specifically, UIM leverages multiple types of user interaction to avoid the ambiguity of multiple matting targets. We show that UIM achieves state-of-the-art performance on the Composition-1K test set and a synthetic unified dataset.
  arXiv  Detail & Related papers  (2022-05-17T13:20:30Z)
• LayoutBERT: Masked Language Layout Model for Object Insertion [3.4806267677524896]
  We propose layoutBERT for the object insertion task. It uses a novel self-supervised masked language model objective and bidirectional multi-head self-attention. We provide both qualitative and quantitative evaluations on datasets from diverse domains.
  arXiv  Detail & Related papers  (2022-04-30T21:35:38Z)
• Generating Object Stamps [47.20601520671103]
  We present an algorithm to generate diverse foreground objects and composite them into background images using a GAN architecture. Our results on the challenging COCO dataset show improved overall quality and diversity compared to state-of-the-art object insertion approaches.
  arXiv  Detail & Related papers  (2020-01-01T14:36:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.