Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning

• URL: http://arxiv.org/abs/2405.02596v1
• Date: Sat, 4 May 2024 07:44:18 GMT
• Title: Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning
• Authors: Jing Xu, Jingzhao Zhang,
• Abstract summary: Fine-tuning large language models (LLM) can be costly. PEFT addresses the problems by training a fraction of the parameters, whose success reveals the expressiveness and flexibility of pretrained models. This paper studies the limit of PEFT, by further simplifying its design and reducing the number of trainable parameters beyond standard setups. We show that Random Masking is surprisingly effective: with a larger-than-expected learning rate, Random Masking can match the performance of standard PEFT algorithms on various tasks, using fewer trainable parameters.
• Score: 17.638387297838936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-tuning large language models (LLM) can be costly. Parameter-efficient fine-tuning (PEFT) addresses the problems by training a fraction of the parameters, whose success reveals the expressiveness and flexibility of pretrained models. This paper studies the limit of PEFT, by further simplifying its design and reducing the number of trainable parameters beyond standard setups. To this end, we use Random Masking to fine-tune the pretrained model. Despite its simplicity, we show that Random Masking is surprisingly effective: with a larger-than-expected learning rate, Random Masking can match the performance of standard PEFT algorithms such as LoRA on various tasks, using fewer trainable parameters. We provide both empirical and theoretical explorations into the success of Random Masking. We show that masking induces a flatter loss landscape and more distant solutions, which allows for and necessitates large learning rates.

Related papers

• MaskLLM: Learnable Semi-Structured Sparsity for Large Language Models [91.4190318047519]
  This work introduces MaskLLM, a learnable pruning method that establishes Semi-structured (or N:M'') Sparsity in Large Language Models. MaskLLM explicitly models N:M patterns as a learnable distribution through Gumbel Softmax sampling.
  arXiv  Detail & Related papers  (2024-09-26T02:37:41Z)
• Masks, Signs, And Learning Rate Rewinding [21.245849787139655]
  Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) We conduct experiments that disentangle the effect of mask learning and parameter optimization. In support of this hypothesis, we prove in a simplified single hidden neuron setting that LRR succeeds in more cases than IMP.
  arXiv  Detail & Related papers  (2024-02-29T15:32:02Z)
• MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning [71.50432879573614]
  Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional. We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank. Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks.
  arXiv  Detail & Related papers  (2024-02-27T07:14:12Z)
• Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection [29.375460634415806]
  We train a large model and then remove its redundant neurons or weights by pruning. This approach achieves state-of-the-art performance across eight datasets from GLUE.
  arXiv  Detail & Related papers  (2023-10-19T22:32:51Z)
• Regularized Mask Tuning: Uncovering Hidden Knowledge in Pre-trained Vision-Language Models [89.07925369856139]
  We design a new type of tuning method, termed as regularized mask tuning, which masks the network parameters through a learnable selection. Inspired by neural pathways, we argue that the knowledge required by a downstream task already exists in the pre-trained weights but just gets concealed in the upstream pre-training stage. It is noteworthy that we manage to deliver 18.73% performance improvement compared to the zero-shot CLIP via masking an average of only 2.56% parameters.
  arXiv  Detail & Related papers  (2023-07-27T17:56:05Z)
• Parameter-Efficient Fine-Tuning without Introducing New Latency [7.631596468553607]
  We introduce a novel adapter technique that directly applies the adapter to pre-trained parameters instead of the hidden representation. Our proposed method attains a new state-of-the-art outcome in terms of both performance and storage efficiency, storing only 0.03% parameters of full fine-tuning.
  arXiv  Detail & Related papers  (2023-05-26T08:44:42Z)
• Masked Autoencoding for Scalable and Generalizable Decision Making [93.84855114717062]
  MaskDP is a simple and scalable self-supervised pretraining method for reinforcement learning and behavioral cloning. We find that a MaskDP model gains the capability of zero-shot transfer to new BC tasks, such as single and multiple goal reaching.
  arXiv  Detail & Related papers  (2022-11-23T07:04:41Z)
• Training Neural Networks with Fixed Sparse Masks [19.58969772430058]
  Recent work has shown that it is possible to update only a small subset of the model's parameters during training. We show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations.
  arXiv  Detail & Related papers  (2021-11-18T18:06:01Z)
• Masksembles for Uncertainty Estimation [60.400102501013784]
  Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expensive to train and evaluate. MC-Dropout is another popular alternative, which is less expensive, but also less reliable.
  arXiv  Detail & Related papers  (2020-12-15T14:39:57Z)
• Masking as an Efficient Alternative to Finetuning for Pretrained Language Models [49.64561153284428]
  We learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. In intrinsic evaluations, we show that representations computed by masked language models encode information necessary for solving downstream tasks.
  arXiv  Detail & Related papers  (2020-04-26T15:03:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.