Adapt Anything: Tailor Any Image Classifiers across Domains And Categories Using Text-to-Image Diffusion Models

• URL: http://arxiv.org/abs/2310.16573v1
• Date: Wed, 25 Oct 2023 11:58:14 GMT
• Title: Adapt Anything: Tailor Any Image Classifiers across Domains And Categories Using Text-to-Image Diffusion Models
• Authors: Weijie Chen, Haoyu Wang, Shicai Yang, Lei Zhang, Wei Wei, Yanning Zhang, Luojun Lin, Di Xie, Yueting Zhuang
• Abstract summary: We aim to study if a modern text-to-image diffusion model can tailor any task-adaptive image classifier across domains and categories. We utilize only one off-the-shelf text-to-image model to synthesize images with category labels derived from the corresponding text prompts.
• Score: 82.95591765009105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We do not pursue a novel method in this paper, but aim to study if a modern text-to-image diffusion model can tailor any task-adaptive image classifier across domains and categories. Existing domain adaptive image classification works exploit both source and target data for domain alignment so as to transfer the knowledge learned from the labeled source data to the unlabeled target data. However, as the development of the text-to-image diffusion model, we wonder if the high-fidelity synthetic data from the text-to-image generator can serve as a surrogate of the source data in real world. In this way, we do not need to collect and annotate the source data for each domain adaptation task in a one-for-one manner. Instead, we utilize only one off-the-shelf text-to-image model to synthesize images with category labels derived from the corresponding text prompts, and then leverage the surrogate data as a bridge to transfer the knowledge embedded in the task-agnostic text-to-image generator to the task-oriented image classifier via domain adaptation. Such a one-for-all adaptation paradigm allows us to adapt anything in the world using only one text-to-image generator as well as the corresponding unlabeled target data. Extensive experiments validate the feasibility of the proposed idea, which even surpasses the state-of-the-art domain adaptation works using the source data collected and annotated in real world.

Related papers

• Generating Reliable Pixel-Level Labels for Source Free Domain Adaptation [13.913151437401472]
  ReGEN comprises an image-to-image translation network and a segmentation network. Our workflow generates target-like images using the noisy predictions from the original target domain images.
  arXiv  Detail & Related papers  (2023-07-03T09:44:13Z)
• iEdit: Localised Text-guided Image Editing with Weak Supervision [53.082196061014734]
  We propose a novel learning method for text-guided image editing. It generates images conditioned on a source image and a textual edit prompt. It shows favourable results against its counterparts in terms of image fidelity, CLIP alignment score and qualitatively for editing both generated and real images.
  arXiv  Detail & Related papers  (2023-05-10T07:39:14Z)
• One-shot Unsupervised Domain Adaptation with Personalized Diffusion Models [15.590759602379517]
  Adapting a segmentation model from a labeled source domain to a target domain is one of the most challenging problems in domain adaptation. We leverage text-to-image diffusion models to generate a synthetic target dataset with photo-realistic images. Experiments show that our method surpasses the state-of-the-art OSUDA methods by up to +7.1%.
  arXiv  Detail & Related papers  (2023-03-31T14:16:38Z)
• Discriminative Class Tokens for Text-to-Image Diffusion Models [107.98436819341592]
  We propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier.
  arXiv  Detail & Related papers  (2023-03-30T05:25:20Z)
• Universal Domain Adaptation for Remote Sensing Image Scene Classification [27.422845844752338]
  We propose a practical universal domain adaptation setting for remote sensing image scene classification. A novel universal domain adaptation method without source data is proposed for cases when the source data is unavailable. Empirical results show that the proposed model is effective and practical for remote sensing image scene classification.
  arXiv  Detail & Related papers  (2023-01-26T20:04:24Z)
• I2F: A Unified Image-to-Feature Approach for Domain Adaptive Semantic Segmentation [55.633859439375044]
  Unsupervised domain adaptation (UDA) for semantic segmentation is a promising task freeing people from heavy annotation work. Key idea to tackle this problem is to perform both image-level and feature-level adaptation jointly. This paper proposes a novel UDA pipeline for semantic segmentation that unifies image-level and feature-level adaptation.
  arXiv  Detail & Related papers  (2023-01-03T15:19:48Z)
• Instance-level Heterogeneous Domain Adaptation for Limited-labeled Sketch-to-Photo Retrieval [36.32367182571164]
  We propose an Instance-level Heterogeneous Domain Adaptation (IHDA) framework. We apply the fine-tuning strategy for identity label learning, aiming to transfer the instance-level knowledge in an inductive transfer manner. Experiments show that our method has set a new state of the art on three sketch-to-photo image retrieval benchmarks without extra annotations.
  arXiv  Detail & Related papers  (2022-11-26T08:50:08Z)
• Source-Free Domain Adaptive Fundus Image Segmentation with Denoised Pseudo-Labeling [56.98020855107174]
  Domain adaptation typically requires to access source domain data to utilize their distribution information for domain alignment with the target data. In many real-world scenarios, the source data may not be accessible during the model adaptation in the target domain due to privacy issue. We present a novel denoised pseudo-labeling method for this problem, which effectively makes use of the source model and unlabeled target data.
  arXiv  Detail & Related papers  (2021-09-19T06:38:21Z)
• Semi-Supervised Domain Adaptation with Prototypical Alignment and Consistency Learning [86.6929930921905]
  This paper studies how much it can help address domain shifts if we further have a few target samples labeled. To explore the full potential of landmarks, we incorporate a prototypical alignment (PA) module which calculates a target prototype for each class from the landmarks. Specifically, we severely perturb the labeled images, making PA non-trivial to achieve and thus promoting model generalizability.
  arXiv  Detail & Related papers  (2021-04-19T08:46:08Z)
• Consistency Regularization with High-dimensional Non-adversarial Source-guided Perturbation for Unsupervised Domain Adaptation in Segmentation [15.428323201750144]
  BiSIDA employs consistency regularization to efficiently exploit information from the unlabeled target dataset. BiSIDA achieves new state-of-the-art on two commonly-used synthetic-to-real domain adaptation benchmarks.
  arXiv  Detail & Related papers  (2020-09-18T03:26:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.