One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data

• URL: http://arxiv.org/abs/2111.12386v1
• Date: Wed, 24 Nov 2021 10:10:05 GMT
• Title: One to Transfer All: A Universal Transfer Framework for Vision Foundation Model with Few Data
• Authors: Yujie Wang, Junqin Huang, Mengya Gao, Yichao Wu, Zhenfei Yin, Ding Liang, Junjie Yan
• Abstract summary: We propose a universal transfer framework: One to Transfer All (OTA) to transfer any Vision Foundation Model (VFM) to any downstream tasks with few downstream data. OTA has no dependency on upstream data, VFM, and downstream tasks when transferring. Massive experiments validate the effectiveness and superiority of our methods in few data setting.
• Score: 56.14205030170083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The foundation model is not the last chapter of the model production pipeline. Transferring with few data in a general way to thousands of downstream tasks is becoming a trend of the foundation model's application. In this paper, we proposed a universal transfer framework: One to Transfer All (OTA) to transfer any Vision Foundation Model (VFM) to any downstream tasks with few downstream data. We first transfer a VFM to a task-specific model by Image Re-representation Fine-tuning (IRF) then distilling knowledge from a task-specific model to a deployed model with data produced by Downstream Image-Guided Generation (DIGG). OTA has no dependency on upstream data, VFM, and downstream tasks when transferring. It also provides a way for VFM researchers to release their upstream information for better transferring but not leaking data due to privacy requirements. Massive experiments validate the effectiveness and superiority of our methods in few data setting. Our code will be released.

Related papers

• Transferring Knowledge from Large Foundation Models to Small Downstream Models [40.38657103236168]
  We introduce Adaptive Feature Transfer (AFT) to transfer knowledge between pre-trained models. AFT operates purely on features, decoupling the choice of the pre-trained model from the smaller downstream model. AFT achieves significantly better downstream performance compared to alternatives with a similar computational cost.
  arXiv  Detail & Related papers  (2024-06-11T15:06:15Z)
• Ambient Diffusion Posterior Sampling: Solving Inverse Problems with Diffusion Models trained on Corrupted Data [56.81246107125692]
  Ambient Diffusion Posterior Sampling (A-DPS) is a generative model pre-trained on one type of corruption. We show that A-DPS can sometimes outperform models trained on clean data for several image restoration tasks in both speed and performance. We extend the Ambient Diffusion framework to train MRI models with access only to Fourier subsampled multi-coil MRI measurements.
  arXiv  Detail & Related papers  (2024-03-13T17:28:20Z)
• Towards Efficient Task-Driven Model Reprogramming with Foundation Models [52.411508216448716]
  Vision foundation models exhibit impressive power, benefiting from the extremely large model capacity and broad training data. However, in practice, downstream scenarios may only support a small model due to the limited computational resources or efficiency considerations. This brings a critical challenge for the real-world application of foundation models: one has to transfer the knowledge of a foundation model to the downstream task.
  arXiv  Detail & Related papers  (2023-04-05T07:28:33Z)
• When does Bias Transfer in Transfer Learning? [89.22641454588278]
  Using transfer learning to adapt a pre-trained "source model" to a downstream "target task" can dramatically increase performance with seemingly no downside. We demonstrate that there can exist a downside after all: bias transfer, or the tendency for biases of the source model to persist even after adapting the model to the target class.
  arXiv  Detail & Related papers  (2022-07-06T17:58:07Z)
• InPars: Data Augmentation for Information Retrieval using Large Language Models [5.851846467503597]
  In this work, we harness the few-shot capabilities of large pretrained language models as synthetic data generators for information retrieval tasks. We show that models finetuned solely on our unsupervised dataset outperform strong baselines such as BM25. retrievers finetuned on both supervised and our synthetic data achieve better zero-shot transfer than models finetuned only on supervised data.
  arXiv  Detail & Related papers  (2022-02-10T16:52:45Z)
• How Well Do Sparse Imagenet Models Transfer? [75.98123173154605]
  Transfer learning is a classic paradigm by which models pretrained on large "upstream" datasets are adapted to yield good results on "downstream" datasets. In this work, we perform an in-depth investigation of this phenomenon in the context of convolutional neural networks (CNNs) trained on the ImageNet dataset. We show that sparse models can match or even outperform the transfer performance of dense models, even at high sparsities.
  arXiv  Detail & Related papers  (2021-11-26T11:58:51Z)
• Progressive and Aligned Pose Attention Transfer for Person Image Generation [59.87492938953545]
  This paper proposes a new generative adversarial network for pose transfer, i.e., transferring the pose of a given person to a target pose. We use two types of blocks, namely Pose-Attentional Transfer Block (PATB) and Aligned Pose-Attentional Transfer Bloc (APATB) We verify the efficacy of the model on the Market-1501 and DeepFashion datasets, using quantitative and qualitative measures.
  arXiv  Detail & Related papers  (2021-03-22T07:24:57Z)
• An Improved Transfer Model: Randomized Transferable Machine [32.50263074872975]
  We propose a new transfer model called Randomized Transferable Machine (RTM) to handle small divergence of domains. Specifically, we work on the new source and target data learned from existing feature-based transfer methods. In principle, the more corruptions are made, the higher the probability of the new target data can be covered by the constructed source data populations.
  arXiv  Detail & Related papers  (2020-11-27T09:37:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.