AnyDoor: Zero-shot Object-level Image Customization

• URL: http://arxiv.org/abs/2307.09481v2
• Date: Wed, 8 May 2024 03:21:34 GMT
• Title: AnyDoor: Zero-shot Object-level Image Customization
• Authors: Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, Hengshuang Zhao,
• Abstract summary: This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations. Our model is trained only once and effortlessly generalizes to diverse object-scene combinations at the inference stage.
• Score: 63.44307304097742
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work presents AnyDoor, a diffusion-based image generator with the power to teleport target objects to new scenes at user-specified locations in a harmonious way. Instead of tuning parameters for each object, our model is trained only once and effortlessly generalizes to diverse object-scene combinations at the inference stage. Such a challenging zero-shot setting requires an adequate characterization of a certain object. To this end, we complement the commonly used identity feature with detail features, which are carefully designed to maintain texture details yet allow versatile local variations (e.g., lighting, orientation, posture, etc.), supporting the object in favorably blending with different surroundings. We further propose to borrow knowledge from video datasets, where we can observe various forms (i.e., along the time axis) of a single object, leading to stronger model generalizability and robustness. Extensive experiments demonstrate the superiority of our approach over existing alternatives as well as its great potential in real-world applications, such as virtual try-on and object moving. Project page is https://damo-vilab.github.io/AnyDoor-Page/.

Related papers

• Generalizable Single-view Object Pose Estimation by Two-side Generating and Matching [19.730504197461144]
  We present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Our method offers generalization to unseen objects without extensive training, operates with a single reference image of the object, and eliminates the need for 3D object models or multiple views of the object.
  arXiv  Detail & Related papers  (2024-11-24T14:31:50Z)
• Learning Global Object-Centric Representations via Disentangled Slot Attention [38.78205074748021]
  This paper introduces a novel object-centric learning method to empower AI systems with human-like capabilities to identify objects across scenes and generate diverse scenes containing specific objects by learning a set of global object-centric representations. Experimental results substantiate the efficacy of the proposed method, demonstrating remarkable proficiency in global object-centric representation learning, object identification, scene generation with specific objects and scene decomposition.
  arXiv  Detail & Related papers  (2024-10-24T14:57:00Z)
• ROAM: Robust and Object-Aware Motion Generation Using Neural Pose Descriptors [73.26004792375556]
  This paper shows that robustness and generalisation to novel scene objects in 3D object-aware character synthesis can be achieved by training a motion model with as few as one reference object. We leverage an implicit feature representation trained on object-only datasets, which encodes an SE(3)-equivariant descriptor field around the object. We demonstrate substantial improvements in 3D virtual character motion and interaction quality and robustness to scenarios with unseen objects.
  arXiv  Detail & Related papers  (2023-08-24T17:59:51Z)
• Anything-3D: Towards Single-view Anything Reconstruction in the Wild [61.090129285205805]
  We introduce Anything-3D, a methodical framework that ingeniously combines a series of visual-language models and the Segment-Anything object segmentation model. Our approach employs a BLIP model to generate textural descriptions, utilize the Segment-Anything model for the effective extraction of objects of interest, and leverages a text-to-image diffusion model to lift object into a neural radiance field.
  arXiv  Detail & Related papers  (2023-04-19T16:39:51Z)
• TopNet: Transformer-based Object Placement Network for Image Compositing [43.14411954867784]
  Local clues in background images are important to determine the compatibility of placing objects with certain locations/scales. We propose to learn the correlation between object features and all local background features with a transformer module. Our new formulation generates a 3D heatmap indicating the plausibility of all location/scale combinations in one network forward pass.
  arXiv  Detail & Related papers  (2023-04-06T20:58:49Z)
• MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare [84.80956484848505]
  MegaPose is a method to estimate the 6D pose of novel objects, that is, objects unseen during training. We present a 6D pose refiner based on a render&compare strategy which can be applied to novel objects. Second, we introduce a novel approach for coarse pose estimation which leverages a network trained to classify whether the pose error between a synthetic rendering and an observed image of the same object can be corrected by the refiner.
  arXiv  Detail & Related papers  (2022-12-13T19:30:03Z)
• Discovering Objects that Can Move [55.743225595012966]
  We study the problem of object discovery -- separating objects from the background without manual labels. Existing approaches utilize appearance cues, such as color, texture, and location, to group pixels into object-like regions. We choose to focus on dynamic objects -- entities that can move independently in the world.
  arXiv  Detail & Related papers  (2022-03-18T21:13:56Z)
• Understanding Object Dynamics for Interactive Image-to-Video Synthesis [8.17925295907622]
  We present an approach that learns naturally-looking global articulations caused by a local manipulation at a pixel level. Our generative model learns to infer natural object dynamics as a response to user interaction. In contrast to existing work on video prediction, we do not synthesize arbitrary realistic videos.
  arXiv  Detail & Related papers  (2021-06-21T17:57:39Z)
• Object-Centric Image Generation from Layouts [93.10217725729468]
  We develop a layout-to-image-generation method to generate complex scenes with multiple objects. Our method learns representations of the spatial relationships between objects in the scene, which lead to our model's improved layout-fidelity. We introduce SceneFID, an object-centric adaptation of the popular Fr'echet Inception Distance metric, that is better suited for multi-object images.
  arXiv  Detail & Related papers  (2020-03-16T21:40:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.