Personalized LoRA for Human-Centered Text Understanding
- URL: http://arxiv.org/abs/2403.06208v1
- Date: Sun, 10 Mar 2024 13:04:54 GMT
- Title: Personalized LoRA for Human-Centered Text Understanding
- Authors: You Zhang, Jin Wang, Liang-Chih Yu, Dan Xu, Xuejie Zhang
- Abstract summary: We introduce personalized LoRA (PLoRA) with a plug-and-play framework for the HCTU task.
PLoRA is effective, parameter-efficient and dynamically deploying in PLMs.
Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios.
- Score: 15.704545145736676
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Effectively and efficiently adapting a pre-trained language model (PLM) for
human-centered text understanding (HCTU) is challenging since user tokens are
million-level in most personalized applications and do not have concrete
explicit semantics. A standard and parameter-efficient approach (e.g., LoRA)
necessitates memorizing numerous suits of adapters for each user. In this work,
we introduce a personalized LoRA (PLoRA) with a plug-and-play (PnP) framework
for the HCTU task. PLoRA is effective, parameter-efficient, and dynamically
deploying in PLMs. Moreover, a personalized dropout and a mutual information
maximizing strategies are adopted and hence the proposed PLoRA can be well
adapted to few/zero-shot learning scenarios for the cold-start issue.
Experiments conducted on four benchmark datasets show that the proposed method
outperforms existing methods in full/few/zero-shot learning scenarios for the
HCTU task, even though it has fewer trainable parameters. For reproducibility,
the code for this paper is available at: https://github.com/yoyo-yun/PLoRA.
Related papers
- MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning [71.50432879573614]
Low-rank adaptation (LoRA) is based on the idea that the adaptation process is intrinsically low-dimensional.
We present MELoRA, a mini-ensemble low-rank adapters that uses fewer trainable parameters while maintaining a higher rank.
Our experimental results show that, compared to LoRA, MELoRA achieves better performance with 8 times fewer trainable parameters on natural language understanding tasks and 36 times fewer trainable parameters on instruction following tasks.
arXiv Detail & Related papers (2024-02-27T07:14:12Z) - PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation [65.268245109828]
We introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process.
We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
arXiv Detail & Related papers (2024-01-20T20:25:17Z) - Efficient slot labelling [0.0]
Slot labelling is an essential component of any dialogue system, aiming to find important arguments in every user turn.
Common approaches involve large pre-trained language models (PLMs) like BERT or RoBERTa, but they face challenges such as high computational requirements and dependence on pre-training data.
We propose a lightweight method which performs on par or better than the state-of-the-art PLM-based methods, while having almost 10x less trainable parameters.
arXiv Detail & Related papers (2024-01-17T17:08:36Z) - Chain of LoRA: Efficient Fine-tuning of Language Models via Residual
Learning [31.036465632204663]
We introduce Chain of LoRA, an iterative optimization framework inspired by the Frank-Wolfe algorithm.
We demonstrate that COLA can consistently outperform LoRA without additional computational or memory costs.
arXiv Detail & Related papers (2024-01-08T14:26:49Z) - PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learning [41.984652077669104]
Experimental results on standard datasets indicate that our method outperforms the state-of-the-art approaches significantly.
Our method exhibits strong robustness and superiority in different settings and degrees of data heterogeneity.
arXiv Detail & Related papers (2024-01-04T06:46:19Z) - Re-parameterized Low-rank Prompt: Generalize a Vision-Language Model
within 0.5K Parameters [75.28536311904489]
We develop a new type of prompt, Re- parameterized Low-rank Prompt (RLP), for both efficient and effective adaptation.
On a series of tasks over 11 datasets, RLP significantly increases the average downstream accuracy of classic prompt tuning by up to 5.25% using merely 0.5K parameters.
arXiv Detail & Related papers (2023-12-17T20:42:43Z) - Sparse Low-rank Adaptation of Pre-trained Language Models [79.74094517030035]
We introduce sparse low-rank adaptation (SoRA) that enables dynamic adjustments to the intrinsic rank during the adaptation process.
Our approach strengthens the representation power of LoRA by initializing it with a higher rank, while efficiently taming a temporarily increased number of parameters.
Our experimental results demonstrate that SoRA can outperform other baselines even with 70% retained parameters and 70% training time.
arXiv Detail & Related papers (2023-11-20T11:56:25Z) - SiRA: Sparse Mixture of Low Rank Adaptation [63.926732717719354]
We investigate the importance of leveraging "sparse" computation and propose SiRA: sparse mixture of low rank.
Specifically it enforces the top $k$ experts routing with a capacity limit restricting the maximum number of tokens each expert can process.
arXiv Detail & Related papers (2023-11-15T18:15:37Z) - IncreLoRA: Incremental Parameter Allocation Method for
Parameter-Efficient Fine-tuning [15.964205804768163]
IncreLoRA is an incremental parameter allocation method that adaptively adds trainable parameters during training.
We conduct extensive experiments on GLUE to demonstrate the effectiveness of IncreLoRA.
arXiv Detail & Related papers (2023-08-23T10:08:10Z) - Evaluating Parameter-Efficient Transfer Learning Approaches on SURE
Benchmark for Speech Understanding [40.27182770995891]
Fine-tuning is widely used as the default algorithm for transfer learning from pre-trained models.
We introduce the Speech UndeRstanding Evaluation (SURE) benchmark for parameter-efficient learning for various speech-processing tasks.
arXiv Detail & Related papers (2023-03-02T08:57:33Z) - Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts.
IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
arXiv Detail & Related papers (2022-06-04T10:08:50Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.