Visual Tuning

• URL: http://arxiv.org/abs/2305.06061v2
• Date: Sat, 13 Apr 2024 06:14:42 GMT
• Title: Visual Tuning
• Authors: Bruce X. B. Yu, Jianlong Chang, Haixin Wang, Lingbo Liu, Shijie Wang, Zhiyu Wang, Junfan Lin, Lingxi Xie, Haojie Li, Zhouchen Lin, Qi Tian, Chang Wen Chen,
• Abstract summary: Fine-tuning visual models has been widely shown promising performance on many downstream visual tasks. Recent advances can achieve superior performance than full-tuning the whole pre-trained parameters. This survey characterizes a large and thoughtful selection of recent works, providing a systematic and comprehensive overview of work and models.
• Score: 143.43997336384126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning visual models has been widely shown promising performance on many downstream visual tasks. With the surprising development of pre-trained visual foundation models, visual tuning jumped out of the standard modus operandi that fine-tunes the whole pre-trained model or just the fully connected layer. Instead, recent advances can achieve superior performance than full-tuning the whole pre-trained parameters by updating far fewer parameters, enabling edge devices and downstream applications to reuse the increasingly large foundation models deployed on the cloud. With the aim of helping researchers get the full picture and future directions of visual tuning, this survey characterizes a large and thoughtful selection of recent works, providing a systematic and comprehensive overview of existing work and models. Specifically, it provides a detailed background of visual tuning and categorizes recent visual tuning techniques into five groups: prompt tuning, adapter tuning, parameter tuning, and remapping tuning. Meanwhile, it offers some exciting research directions for prospective pre-training and various interactions in visual tuning.

Related papers

• Partial Fine-Tuning: A Successor to Full Fine-Tuning for Vision Transformers [50.23439411530435]
  We show that Partial Fine-Tuning can be an innovative and promising direction capable of concurrently enhancing both efficiency and accuracy. We propose a novel fine-tuned angle metric to guide the selection of appropriate layers for partial fine-tuning. Comprehensive experiments on a wide range of datasets and models validate the great potential of partial fine-tuning.
  arXiv  Detail & Related papers  (2023-12-25T10:11:34Z)
• VS-TransGRU: A Novel Transformer-GRU-based Framework Enhanced by Visual-Semantic Fusion for Egocentric Action Anticipation [33.41226268323332]
  Egocentric action anticipation is a challenging task that aims to make advanced predictions of future actions in the first-person view. Most existing methods focus on improving the model architecture and loss function based on the visual input and recurrent neural network. We propose a novel visual-semantic fusion enhanced and Transformer GRU-based action anticipation framework.
  arXiv  Detail & Related papers  (2023-07-08T06:49:54Z)
• MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models [12.397136690734865]
  We propose a novel approach called Multi-modal Deep-symphysis Prompt Tuning, dubbed as MuDPT. MuDPT extends independent multi-modal prompt tuning by learning a model-agnostic transformative network to allow deep hierarchical bi-directional prompt fusion. Compared with the state-of-the-art methods, MuDPT achieves better recognition and generalization ability with an apparent margin.
  arXiv  Detail & Related papers  (2023-06-20T09:15:52Z)
• GPT4Image: Can Large Pre-trained Models Help Vision Models on Perception Tasks? [51.22096780511165]
  We present a new learning paradigm in which the knowledge extracted from large pre-trained models are utilized to help models like CNN and ViT learn enhanced representations. We feed detailed descriptions into a pre-trained encoder to extract text embeddings with rich semantic information that encodes the content of images.
  arXiv  Detail & Related papers  (2023-06-01T14:02:45Z)
• Unified Vision and Language Prompt Learning [86.1530128487077]
  We present a systematic study on two representative prompt tuning methods, namely text prompt tuning and visual prompt tuning. A major finding is that text prompt tuning fails on data with high intra-class visual variances while visual prompt tuning cannot handle low inter-class variances. To combine the best from both worlds, we propose a simple approach called Unified Prompt Tuning (UPT), which essentially learns a tiny neural network to jointly optimize prompts across different modalities.
  arXiv  Detail & Related papers  (2022-10-13T17:50:24Z)
• Prompt Tuning for Generative Multimodal Pretrained Models [75.44457974275154]
  We implement prompt tuning on the unified sequence-to-sequence pretrained model adaptive to both understanding and generation tasks. Experimental results demonstrate that the light-weight prompt tuning can achieve comparable performance with finetuning. In comparison with finetuned models, the prompt-tuned models demonstrate improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2022-08-04T08:56:38Z)
• Pro-tuning: Unified Prompt Tuning for Vision Tasks [133.12978197265596]
  Fine-tuning is the de-facto approach to leverage pre-trained vision models to perform downstream tasks. In this work, we propose parameter-efficient Prompt tuning (Pro-tuning) to adapt frozen vision models to various downstream vision tasks.
  arXiv  Detail & Related papers  (2022-07-28T21:09:31Z)
• Neural Prompt Search [38.68910532066619]
  We propose Neural prOmpt seArcH, a novel approach that learns, for large vision models, the optimal design of prompt modules. NOAH learns, for large vision models, the optimal design of prompt modules through a neural architecture search algorithm.
  arXiv  Detail & Related papers  (2022-06-09T17:59:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.