Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild

• URL: http://arxiv.org/abs/2412.03150v2
• Date: Tue, 18 Mar 2025 07:31:49 GMT
• Title: Appearance Matching Adapter for Exemplar-based Semantic Image Synthesis in-the-Wild
• Authors: Siyoon Jin, Jisu Nam, Jiyoung Kim, Dahyun Chung, Yeong-Seok Kim, Joonhyung Park, Heonjeong Chu, Seungryong Kim,
• Abstract summary: Exemplar-based semantic image synthesis generates images aligned with semantic content while preserving the appearance of an exemplar.<n>Recent tuning-free approaches address this by transferring local appearance via implicit cross-image matching.<n>We propose AM-Adapter to address exemplar-based semantic image synthesis in-the-wild.
• Score: 29.23745176017559
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Exemplar-based semantic image synthesis generates images aligned with semantic content while preserving the appearance of an exemplar. Conventional structure-guidance models like ControlNet, are limited as they rely solely on text prompts to control appearance and cannot utilize exemplar images as input. Recent tuning-free approaches address this by transferring local appearance via implicit cross-image matching in the augmented self-attention mechanism of pre-trained diffusion models. However, prior works are often restricted to single-object cases or foreground object appearance transfer, struggling with complex scenes involving multiple objects. To overcome this, we propose AM-Adapter (Appearance Matching Adapter) to address exemplar-based semantic image synthesis in-the-wild, enabling multi-object appearance transfer from a single scene-level image. AM-Adapter automatically transfers local appearances from the scene-level input. AM-Adapter alternatively provides controllability to map user-defined object details to specific locations in the synthesized images. Our learnable framework enhances cross-image matching within augmented self-attention by integrating semantic information from segmentation maps. To disentangle generation and matching, we adopt stage-wise training. We first train the structure-guidance and generation networks, followed by training the matching adapter while keeping the others frozen. During inference, we introduce an automated exemplar retrieval method for selecting exemplar image-segmentation pairs efficiently. Despite utilizing minimal learnable parameters, AM-Adapter achieves state-of-the-art performance, excelling in both semantic alignment and local appearance fidelity. Extensive ablations validate our design choices. Code and weights will be released.: https://cvlab-kaist.github.io/AM-Adapter/

Related papers

• UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation [64.8341372591993]
  We propose a new approach to unify controllable generation within a single framework. Specifically, we propose the unified image-instruction adapter (UNIC-Adapter) built on the Multi-Modal-Diffusion Transformer architecture. Our UNIC-Adapter effectively extracts multi-modal instruction information by incorporating both conditional images and task instructions.
  arXiv  Detail & Related papers  (2024-12-25T15:19:02Z)
• Object-Attribute Binding in Text-to-Image Generation: Evaluation and Control [58.37323932401379]
  Current diffusion models create images given a text prompt as input but struggle to correctly bind attributes mentioned in the text to the right objects in the image. We propose focused cross-attention (FCA) that controls the visual attention maps by syntactic constraints found in the input sentence. We show substantial improvements in T2I generation and especially its attribute-object binding on several datasets.
  arXiv  Detail & Related papers  (2024-04-21T20:26:46Z)
• Deep Semantic-Visual Alignment for Zero-Shot Remote Sensing Image Scene Classification [26.340737217001497]
  Zero-shot learning (ZSL) allows for identifying novel classes that are not seen during training. Previous ZSL models mainly depend on manually-labeled attributes or word embeddings extracted from language models to transfer knowledge from seen classes to novel classes. We propose to collect visually detectable attributes automatically. We predict attributes for each class by depicting the semantic-visual similarity between attributes and images.
  arXiv  Detail & Related papers  (2024-02-03T09:18:49Z)
• A Transformer-Based Adaptive Semantic Aggregation Method for UAV Visual Geo-Localization [2.1462492411694756]
  This paper addresses the task of Unmanned Aerial Vehicles (UAV) visual geo-localization. Part matching is crucial for UAV visual geo-localization since part-level representations can capture image details and help to understand the semantic information of scenes. We introduce a transformer-based adaptive semantic aggregation method that regards parts as the most representative semantics in an image.
  arXiv  Detail & Related papers  (2024-01-03T06:58:52Z)
• Hyper-VolTran: Fast and Generalizable One-Shot Image to 3D Object Structure via HyperNetworks [53.67497327319569]
  We introduce a novel neural rendering technique to solve image-to-3D from a single view. Our approach employs the signed distance function as the surface representation and incorporates generalizable priors through geometry-encoding volumes and HyperNetworks. Our experiments show the advantages of our proposed approach with consistent results and rapid generation.
  arXiv  Detail & Related papers  (2023-12-24T08:42:37Z)
• Disentangling Structure and Appearance in ViT Feature Space [26.233355454282446]
  We present a method for semantically transferring the visual appearance of one natural image to another. Specifically, our goal is to generate an image in which objects in a source structure image are "painted" with the visual appearance of their semantically related objects in a target appearance image. We propose two frameworks of semantic appearance transfer -- "Splice", which works by training a generator on a single and arbitrary pair of structure-appearance images, and "SpliceNet", a feed-forward real-time appearance transfer model trained on a dataset of images from a specific domain.
  arXiv  Detail & Related papers  (2023-11-20T21:20:15Z)
• Cross-Image Attention for Zero-Shot Appearance Transfer [68.43651329067393]
  We introduce a cross-image attention mechanism that implicitly establishes semantic correspondences across images. We harness three mechanisms that either manipulate the noisy latent codes or the model's internal representations throughout the denoising process. Experiments show that our method is effective across a wide range of object categories and is robust to variations in shape, size, and viewpoint.
  arXiv  Detail & Related papers  (2023-11-06T18:33:24Z)
• Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation [16.863038973001483]
  This work introduces three techniques for diffusion-synthetic semantic segmentation training. First, reliability-aware robust training, originally used in weakly supervised learning, helps segmentation with insufficient synthetic mask quality. Second, large-scale pretraining of whole segmentation models, not only backbones, on synthetic ImageNet-1k-class images with pixel-labels benefits downstream segmentation tasks. Third, we introduce prompt augmentation, data augmentation to the prompt text set to scale up and diversify training images with a limited text resources.
  arXiv  Detail & Related papers  (2023-09-04T05:34:19Z)
• Dense Text-to-Image Generation with Attention Modulation [49.287458275920514]
  Existing text-to-image diffusion models struggle to synthesize realistic images given dense captions. We propose DenseDiffusion, a training-free method that adapts a pre-trained text-to-image model to handle such dense captions. We achieve similar-quality visual results with models specifically trained with layout conditions.
  arXiv  Detail & Related papers  (2023-08-24T17:59:01Z)
• Zero-shot spatial layout conditioning for text-to-image diffusion models [52.24744018240424]
  Large-scale text-to-image diffusion models have significantly improved the state of the art in generative image modelling. We consider image generation from text associated with segments on the image canvas, which combines an intuitive natural language interface with precise spatial control over the generated content. We propose ZestGuide, a zero-shot segmentation guidance approach that can be plugged into pre-trained text-to-image diffusion models.
  arXiv  Detail & Related papers  (2023-06-23T19:24:48Z)
• Masked and Adaptive Transformer for Exemplar Based Image Translation [16.93344592811513]
  Cross-domain semantic matching is challenging. We propose a masked and adaptive transformer (MAT) for learning accurate cross-domain correspondence. We devise a novel contrastive style learning method, for acquire quality-discriminative style representations.
  arXiv  Detail & Related papers  (2023-03-30T03:21:14Z)
• Correlational Image Modeling for Self-Supervised Visual Pre-Training [81.82907503764775]
  Correlational Image Modeling is a novel and surprisingly effective approach to self-supervised visual pre-training. Three key designs enable correlational image modeling as a nontrivial and meaningful self-supervisory task.
  arXiv  Detail & Related papers  (2023-03-22T15:48:23Z)
• Situational Perception Guided Image Matting [16.1897179939677]
  We propose a Situational Perception Guided Image Matting (SPG-IM) method that mitigates subjective bias of matting annotations. SPG-IM can better associate inter-objects and object-to-environment saliency, and compensate the subjective nature of image matting.
  arXiv  Detail & Related papers  (2022-04-20T07:35:51Z)
• Retrieval-based Spatially Adaptive Normalization for Semantic Image Synthesis [68.1281982092765]
  We propose a novel normalization module, termed as REtrieval-based Spatially AdaptIve normaLization (RESAIL) RESAIL provides pixel level fine-grained guidance to the normalization architecture. Experiments on several challenging datasets show that our RESAIL performs favorably against state-of-the-arts in terms of quantitative metrics, visual quality, and subjective evaluation.
  arXiv  Detail & Related papers  (2022-04-06T14:21:39Z)
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters [84.88106370842883]
  We show that there is an alternative path to achieve better vision-language models other than prompt tuning. CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending. Experiments and extensive ablation studies on various visual classification tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2021-10-09T11:39:30Z)
• Image Shape Manipulation from a Single Augmented Training Sample [26.342929563689218]
  DeepSIM is a generative model for conditional image manipulation based on a single image. Our network learns to map between a primitive representation of the image to the image itself.
  arXiv  Detail & Related papers  (2021-09-13T17:44:04Z)
• ADeLA: Automatic Dense Labeling with Attention for Viewpoint Adaptation in Semantic Segmentation [27.69348820877977]
  We describe an unsupervised domain adaptation method for image content shift caused by viewpoint changes for a semantic segmentation task. Our method works without aligning any statistics of the images between the two domains. It utilizes a view transformation network trained only on color images to hallucinate the semantic images for the target.
  arXiv  Detail & Related papers  (2021-07-29T19:10:18Z)
• Graph Sampling Based Deep Metric Learning for Generalizable Person Re-Identification [114.56752624945142]
  We argue that the most popular random sampling method, the well-known PK sampler, is not informative and efficient for deep metric learning. We propose an efficient mini batch sampling method called Graph Sampling (GS) for large-scale metric learning.
  arXiv  Detail & Related papers  (2021-04-04T06:44:15Z)
• Image Shape Manipulation from a Single Augmented Training Sample [24.373900721120286]
  DeepSIM is a generative model for conditional image manipulation based on a single image. Our network learns to map between a primitive representation of the image to the image itself.
  arXiv  Detail & Related papers  (2020-07-02T17:55:27Z)
• Example-Guided Image Synthesis across Arbitrary Scenes using Masked Spatial-Channel Attention and Self-Supervision [83.33283892171562]
  Example-guided image synthesis has recently been attempted to synthesize an image from a semantic label map and an exemplary image. In this paper, we tackle a more challenging and general task, where the exemplar is an arbitrary scene image that is semantically different from the given label map. We propose an end-to-end network for joint global and local feature alignment and synthesis.
  arXiv  Detail & Related papers  (2020-04-18T18:17:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.