Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks

• URL: http://arxiv.org/abs/2403.09377v2
• Date: Fri, 12 Jul 2024 12:54:42 GMT
• Title: Introducing Routing Functions to Vision-Language Parameter-Efficient Fine-Tuning with Low-Rank Bottlenecks
• Authors: Tingyu Qu, Tinne Tuytelaars, Marie-Francine Moens,
• Abstract summary: We propose a family of operations, called routing functions, to enhance vision-language (VL) alignment in low-rank bottlenecks. In various VL PEFT settings, the routing functions significantly improve performance of the original PEFT methods.
• Score: 54.31708859631821
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Mainstream parameter-efficient fine-tuning (PEFT) methods, such as LoRA or Adapter, project a model's hidden states to a lower dimension, allowing pre-trained models to adapt to new data through this low-rank bottleneck. However, PEFT tasks involving multiple modalities, like vision-language (VL) tasks, require not only adaptation to new data but also learning the relationship between different modalities. Targeting at VL PEFT tasks, we propose a family of operations, called routing functions, to enhance VL alignment in the low-rank bottlenecks. These feature routing functions adopt linear operations and do not introduce new trainable parameters. In-depth analyses are conducted to study their behavior. In various VL PEFT settings, the routing functions significantly improve performance of the original PEFT methods, achieving over 20\% improvement on VQAv2 ($\text{RoBERTa}_{\text{large}}$+ViT-L/16) and 30\% on COCO Captioning (GPT2-medium+ViT-L/16). Also when fine-tuning a pre-trained multimodal model such as CLIP-BART, we observe smaller but consistent improvements across a range of VL PEFT tasks. Our code is available at https://github.com/tingyu215/Routing_VLPEFT.

Related papers

• GenFT: A Generative Parameter-Efficient Fine-Tuning Method for Pretrained Foundation Models [23.85784506399565]
  Generative Fine-Tuning (GenFT) is a novel method that extracts structured, transferable information from $Delta W$ for efficient $Delta W$ training.<n>Experiments on VTAB-1K, FGVC, and GLUE benchmarks demonstrate that GenFT outperforms state-of-the-art PEFT methods.
  arXiv  Detail & Related papers  (2025-05-21T08:33:26Z)
• Adaptive parameter-efficient fine-tuning via Hessian-informed subset selection [8.885727065823156]
  We propose a Hessian-informed approach to fine-tuning models.<n>AdaPEFT adapts to various tasks and models, in which the selected subset empirically transfers across training horizons and model sizes.
  arXiv  Detail & Related papers  (2025-05-18T23:45:50Z)
• Visual Fourier Prompt Tuning [63.66866445034855]
  We propose the Visual Fourier Prompt Tuning (VFPT) method as a general and effective solution for adapting large-scale transformer-based models. Our approach incorporates the Fast Fourier Transform into prompt embeddings and harmoniously considers both spatial and frequency domain information. Our results demonstrate that our approach outperforms current state-of-the-art baselines on two benchmarks.
  arXiv  Detail & Related papers  (2024-11-02T18:18:35Z)
• MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning [28.254318215697527]
  Vision-Language models (VLMs) come with high computational costs due to their large number of parameters. Existing techniques for VLMs are task-specific, and thus require pruning the network from scratch for each new task of interest. We explore a new direction: Task-Agnostic Vision-Language Pruning (TA-language) We propose Multimodal FlowPruning (MULTIFLOW), a first, gradient-free, pruning framework for TA-language.
  arXiv  Detail & Related papers  (2024-04-08T15:51:21Z)
• Context-PEFT: Efficient Multi-Modal, Multi-Task Fine-Tuning [12.648711621637663]
  This paper introduces a novel. COCO-Efficient Fine-Tuning (PEFT) framework for multi-modal, multi-task transfer learning with pre-trained language models. We propose Context-PEFT, which learns different groups of adaptor parameters based on the token's domain. Our method is evaluated on the captioning task, where it outperforms full fine-tuning under similar data constraints.
  arXiv  Detail & Related papers  (2023-12-14T13:00:24Z)
• Prototype-based HyperAdapter for Sample-Efficient Multi-task Tuning [30.251155072822055]
  Prototype-based HyperAdapter (PHA) is a novel framework built on the adapter-tuning and hypernetwork. It introduces an instance-dense retriever and prototypical hypernetwork to generate conditional modules in a sample-efficient manner. We show that PHA strikes a better trade-off between trainable parameters, accuracy on stream tasks, and sample efficiency.
  arXiv  Detail & Related papers  (2023-10-18T02:42:17Z)
• Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models [79.34513906324727]
  In this paper, we aim at parameter and efficient transfer learning (PCETL) for vision-language pre-trained models. We propose a novel dynamic architecture skipping (DAS) approach towards effective PCETL.
  arXiv  Detail & Related papers  (2023-09-04T09:34:33Z)
• Sensitivity-Aware Visual Parameter-Efficient Fine-Tuning [91.5113227694443]
  We propose a novel visual. sensuous-aware fine-Tuning (SPT) scheme. SPT allocates trainable parameters to task-specific important positions. Experiments on a wide range of downstream recognition tasks show that our SPT is complementary to the existing PEFT methods.
  arXiv  Detail & Related papers  (2023-03-15T12:34:24Z)
• Pruning Self-attentions into Convolutional Layers in Single Path [89.55361659622305]
  Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. We propose Single-Path Vision Transformer pruning (SPViT) to efficiently and automatically compress the pre-trained ViTs. Our SPViT can trim 52.0% FLOPs for DeiT-B and get an impressive 0.6% top-1 accuracy gain simultaneously.
  arXiv  Detail & Related papers  (2021-11-23T11:35:54Z)
• An Empirical Study of Training End-to-End Vision-and-Language Transformers [50.23532518166621]
  We present METER(textbfMultimodal textbfEnd-to-end textbfTransformtextbfER), through which we investigate how to design and pre-train a fully transformer-based VL model. Specifically, we dissect the model designs along multiple dimensions: vision encoders (e.g., CLIP-ViT, Swin transformer), text encoders (e.g., RoBERTa, DeBERTa), multimodal fusion (e.g., merged attention vs. co-
  arXiv  Detail & Related papers  (2021-11-03T17:55:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.