MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model

• URL: http://arxiv.org/abs/2405.00293v1
• Date: Wed, 1 May 2024 03:15:28 GMT
• Title: MoPEFT: A Mixture-of-PEFTs for the Segment Anything Model
• Authors: Rajat Sahay, Andreas Savakis,
• Abstract summary: Mixture-of-PEFTs methods (MoPEFT) is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emergence of foundation models, such as the Segment Anything Model (SAM), has sparked interest in Parameter-Efficient Fine-Tuning (PEFT) methods that tailor these large models to application domains outside their training data. However, different PEFT techniques modify the representation of a model differently, making it a non-trivial task to select the most appropriate method for the domain of interest. We propose a new framework, Mixture-of-PEFTs methods (MoPEFT), that is inspired by traditional Mixture-of-Experts (MoE) methodologies and is utilized for fine-tuning SAM. Our MoPEFT framework incorporates three different PEFT techniques as submodules and dynamically learns to activate the ones that are best suited for a given data-task setup. We test our method on the Segment Anything Model and show that MoPEFT consistently outperforms other fine-tuning methods on the MESS benchmark.

Related papers

• PERFT: Parameter-Efficient Routed Fine-Tuning for Mixture-of-Expert Model [30.620582168350698]
  Mixture-of-Experts (MoE) has emerged as a powerful approach for scaling transformers with improved resource utilization. Inspired by recent works on -Efficient Fine-Tuning (PEFT), we present a unified framework for integrating PEFT modules directly into the MoE mechanism.
  arXiv  Detail & Related papers  (2024-11-12T22:03:37Z)
• Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging [111.8456671452411]
  Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. We propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. We show that WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.
  arXiv  Detail & Related papers  (2024-10-29T07:16:31Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Is Modularity Transferable? A Case Study through the Lens of Knowledge Distillation [59.37775534633868]
  We present an extremely straightforward approach to transferring pre-trained, task-specific PEFT modules between same-family PLMs. We also propose a method that allows the transfer of modules between incompatible PLMs without any change in the inference complexity.
  arXiv  Detail & Related papers  (2024-03-27T17:50:00Z)
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
  Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
  arXiv  Detail & Related papers  (2024-02-01T08:58:57Z)
• Assessing the Portability of Parameter Matrices Trained by Parameter-Efficient Finetuning Methods [6.653947064461629]
  We investigate the inverse: porting whole functional modules that encode task-specific knowledge from one model to another. We designed a study comprising 1,440 training/testing runs to test the portability of modules trained by parameter-efficient finetuning (PEFT) techniques. We find that the ported modules far outperform the two alternatives tested, but that there are interesting performance differences between the four PEFT techniques.
  arXiv  Detail & Related papers  (2024-01-25T15:11:07Z)
• ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantization [100.90624220423634]
  We present ComPEFT, a novel method for compressing fine-tuning residuals (task vectors) of PEFT based models. In extensive evaluation across T5, T0, and LLaMA-based models with 200M - 65B parameters, ComPEFT achieves compression ratios of 8x - 50x.
  arXiv  Detail & Related papers  (2023-11-22T05:28:59Z)
• Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling [42.42235704360381]
  Large language models (LLMs) and vision language models (VLMs) demonstrate excellent performance on a wide range of tasks. These large scales make it impossible to adapt and deploy fully specialized models given a task of interest. In this work, we describe AdaLink as a non-intrusive PEFT technique that achieves competitive performance.
  arXiv  Detail & Related papers  (2023-10-18T16:43:08Z)
• UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuning [64.638804236566]
  We propose a unified framework, UniPELT, which incorporates different PELT methods as submodules and learns to activate the ones that best suit the current data or task setup. Remarkably, on the GLUE benchmark, UniPELT consistently achieves 13pt gains compared to the best individual PELT method that it incorporates and even outperforms fine-tuning under different setups.
  arXiv  Detail & Related papers  (2021-10-14T17:40:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.