Vision-Language Dataset Distillation

• URL: http://arxiv.org/abs/2308.07545v4
• Date: Tue, 20 Aug 2024 14:59:55 GMT
• Title: Vision-Language Dataset Distillation
• Authors: Xindi Wu, Byron Zhang, Zhiwei Deng, Olga Russakovsky,
• Abstract summary: We design the first vision-language dataset distillation method, building on the idea of trajectory matching. A key challenge is that vision-language datasets do not have a set of discrete classes. Our proposed method jointly distills image-text pairs in a contrastive formulation.
• Score: 26.886260846439612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dataset distillation methods reduce large-scale datasets to smaller sets of synthetic data, preserving sufficient information to quickly train a new model from scratch. However, prior work on dataset distillation has focused exclusively on image classification datasets, whereas modern large-scale datasets are primarily vision-language datasets. In this work, we design the first vision-language dataset distillation method, building on the idea of trajectory matching. A key challenge is that vision-language datasets do not have a set of discrete classes. To overcome this, our proposed method jointly distills image-text pairs in a contrastive formulation. Further, we leverage Low-Rank Adaptation (LoRA) matching to enable more efficient and effective trajectory matching in complex modern vision-language models. Since there are no existing baselines, we compare our distillation approach with three adapted vision-language coreset selection methods. We demonstrate significant improvements on the challenging Flickr30K and COCO retrieval benchmarks: for example, on Flickr30K, the best coreset selection method selecting 1000 image-text pairs for training achieves only 5.6% image-to-text retrieval accuracy (i.e., recall@1); in contrast, our dataset distillation almost doubles that to 9.9% with just 100 training pairs, an order of magnitude fewer.

Related papers

• Low-Rank Similarity Mining for Multimodal Dataset Distillation [50.45577048854653]
  We propose Low-Rank Similarity Mining (LoRS) for multimodal dataset distillation. LoRS distills a ground truth similarity matrix with image-text pairs, and leverages low-rank factorization for efficiency and scalability.
  arXiv  Detail & Related papers  (2024-06-06T07:05:20Z)
• Dataset Distillation via Adversarial Prediction Matching [24.487950991247764]
  We propose an adversarial framework to solve the dataset distillation problem efficiently. Our method can produce synthetic datasets just 10% the size of the original, yet achieve, on average, 94% of the test accuracy of models trained on the full original datasets.
  arXiv  Detail & Related papers  (2023-12-14T13:19:33Z)
• DataDAM: Efficient Dataset Distillation with Attention Matching [15.300968899043498]
  Researchers have long tried to minimize training costs in deep learning by maintaining strong generalization across diverse datasets. Emerging research on dataset aims to reduce training costs by creating a small synthetic set that contains the information of a larger real dataset. However, the synthetic data generated by previous methods are not guaranteed to distribute and discriminate as well as the original training data.
  arXiv  Detail & Related papers  (2023-09-29T19:07:48Z)
• Generalizing Dataset Distillation via Deep Generative Prior [75.9031209877651]
  We propose to distill an entire dataset's knowledge into a few synthetic images. The idea is to synthesize a small number of synthetic data points that, when given to a learning algorithm as training data, result in a model approximating one trained on the original data. We present a new optimization algorithm that distills a large number of images into a few intermediate feature vectors in the generative model's latent space.
  arXiv  Detail & Related papers  (2023-05-02T17:59:31Z)
• Semi-Supervised Image Captioning by Adversarially Propagating Labeled Data [95.0476489266988]
  We present a novel data-efficient semi-supervised framework to improve the generalization of image captioning models. Our proposed method trains a captioner to learn from a paired data and to progressively associate unpaired data. Our extensive and comprehensive empirical results both on (1) image-based and (2) dense region-based captioning datasets followed by comprehensive analysis on the scarcely-paired dataset.
  arXiv  Detail & Related papers  (2023-01-26T15:25:43Z)
• Ensemble Transfer Learning for Multilingual Coreference Resolution [60.409789753164944]
  A problem that frequently occurs when working with a non-English language is the scarcity of annotated training data. We design a simple but effective ensemble-based framework that combines various transfer learning techniques. We also propose a low-cost TL method that bootstraps coreference resolution models by utilizing Wikipedia anchor texts.
  arXiv  Detail & Related papers  (2023-01-22T18:22:55Z)
• Improving Zero-shot Generalization and Robustness of Multi-modal Models [70.14692320804178]
  Multi-modal image-text models such as CLIP and LiT have demonstrated impressive performance on image classification benchmarks. We investigate the reasons for this performance gap and find that many of the failure cases are caused by ambiguity in the text prompts. We propose a simple and efficient way to improve accuracy on such uncertain images by making use of the WordNet hierarchy.
  arXiv  Detail & Related papers  (2022-12-04T07:26:24Z)
• CCMB: A Large-scale Chinese Cross-modal Benchmark [46.349966178044184]
  We build a large-scale high-quality Chinese Cross-Modal Benchmark named CCMB for the research community. Zero contains 250 million images paired with 750 million text descriptions, plus two of the five fine-tuning datasets are also the largest ones for Chinese cross-modal downstream tasks.
  arXiv  Detail & Related papers  (2022-05-08T13:19:23Z)
• Data Efficient Language-supervised Zero-shot Recognition with Optimal Transport Distillation [43.03533959429743]
  We propose OTTER, which uses online optimal transport to find a soft image-text match as labels for contrastive learning. Based on pretrained image and text encoders, models trained with OTTER achieve strong performance with only 3M image text pairs.
  arXiv  Detail & Related papers  (2021-12-17T11:27:26Z)
• Efficient Conditional Pre-training for Transfer Learning [71.01129334495553]
  We propose efficient filtering methods to select relevant subsets from the pre-training dataset. We validate our techniques by pre-training on ImageNet in both the unsupervised and supervised settings. We improve standard ImageNet pre-training by 1-3% by tuning available models on our subsets and pre-training on a dataset filtered from a larger scale dataset.
  arXiv  Detail & Related papers  (2020-11-20T06:16:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.