Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation

• URL: http://arxiv.org/abs/2305.16289v2
• Date: Sun, 29 Oct 2023 22:52:04 GMT
• Title: Diversify Your Vision Datasets with Automatic Diffusion-Based Augmentation
• Authors: Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E. Gonzalez, Trevor Darrell
• Abstract summary: ALIA (Automated Language-guided Image Augmentation) is a method which utilizes large vision and language models to automatically generate natural language descriptions of a dataset's domains. To maintain data integrity, a model trained on the original dataset filters out minimal image edits and those which corrupt class-relevant information. We show that ALIA is able to surpasses traditional data augmentation and text-to-image generated data on fine-grained classification tasks.
• Score: 66.6546668043249
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Many fine-grained classification tasks, like rare animal identification, have limited training data and consequently classifiers trained on these datasets often fail to generalize to variations in the domain like changes in weather or location. As such, we explore how natural language descriptions of the domains seen in training data can be used with large vision models trained on diverse pretraining datasets to generate useful variations of the training data. We introduce ALIA (Automated Language-guided Image Augmentation), a method which utilizes large vision and language models to automatically generate natural language descriptions of a dataset's domains and augment the training data via language-guided image editing. To maintain data integrity, a model trained on the original dataset filters out minimal image edits and those which corrupt class-relevant information. The resulting dataset is visually consistent with the original training data and offers significantly enhanced diversity. We show that ALIA is able to surpasses traditional data augmentation and text-to-image generated data on fine-grained classification tasks, including cases of domain generalization and contextual bias. Code is available at https://github.com/lisadunlap/ALIA.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Regularized Training with Generated Datasets for Name-Only Transfer of Vision-Language Models [36.59260354292177]
  Recent advancements in text-to-image generation have inspired researchers to generate datasets tailored for perception models using generative models. We aim to fine-tune vision-language models to a specific classification model without access to any real images. Despite the high fidelity of generated images, we observed a significant performance degradation when fine-tuning the model using the generated datasets.
  arXiv  Detail & Related papers  (2024-06-08T10:43:49Z)
• WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization [63.98650220772378]
  We present WIDIn, Wording Images for Domain-Invariant representation, to disentangle discriminative visual representation. We first estimate the language embedding with fine-grained alignment, which can be used to adaptively identify and then remove domain-specific counterpart. We show that WIDIn can be applied to both pretrained vision-language models like CLIP, and separately trained uni-modal models like MoCo and BERT.
  arXiv  Detail & Related papers  (2024-05-28T17:46:27Z)
• ASPIRE: Language-Guided Data Augmentation for Improving Robustness Against Spurious Correlations [43.323791505213634]
  ASPIRE (Language-guided Data Augmentation for SPurIous correlation REmoval) is a solution for supplementing the training dataset with images without spurious features. It can generate non-spurious images without requiring any group labeling or existing non-spurious images in the training set. It improves the worst-group classification accuracy of prior methods by 1% - 38%.
  arXiv  Detail & Related papers  (2023-08-19T20:18:15Z)
• Effective Data Augmentation With Diffusion Models [65.09758931804478]
  We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models. Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples. We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.
  arXiv  Detail & Related papers  (2023-02-07T20:42:28Z)
• Using Language to Extend to Unseen Domains [81.37175826824625]
  It is expensive to collect training data for every possible domain that a vision model may encounter when deployed. We consider how simply verbalizing the training domain as well as domains we want to extend to but do not have data for can improve robustness. Using a multimodal model with a joint image and language embedding space, our method LADS learns a transformation of the image embeddings from the training domain to each unseen test domain.
  arXiv  Detail & Related papers  (2022-10-18T01:14:02Z)
• Prefix Conditioning Unifies Language and Label Supervision [84.11127588805138]
  We show that dataset biases negatively affect pre-training by reducing the generalizability of learned representations. In experiments, we show that this simple technique improves the performance in zero-shot image recognition accuracy and robustness to the image-level distribution shift.
  arXiv  Detail & Related papers  (2022-06-02T16:12:26Z)
• Classifying Textual Data with Pre-trained Vision Models through Transfer Learning and Data Transformations [0.0]
  We propose to use knowledge acquired by benchmark Vision Models which are trained on ImageNet to help a much smaller architecture learn to classify text. An analysis of different domains and the Transfer Learning method is carried out. The main contribution of this work is a novel approach which links large pretrained models on both language and vision to achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2021-06-23T15:53:38Z)
• Domain Adaptation with Morphologic Segmentation [8.0698976170854]
  We present a novel domain adaptation framework that uses morphologic segmentation to translate images from arbitrary input domains (real and synthetic) into a uniform output domain. Our goal is to establish a preprocessing step that unifies data from multiple sources into a common representation. We showcase the effectiveness of our approach by qualitatively and quantitatively evaluating our method on four data sets of simulated and real data of urban scenes.
  arXiv  Detail & Related papers  (2020-06-16T17:06:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.