Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models

• URL: http://arxiv.org/abs/2409.09510v1
• Date: Sat, 14 Sep 2024 19:18:26 GMT
• Title: Comparing Retrieval-Augmentation and Parameter-Efficient Fine-Tuning for Privacy-Preserving Personalization of Large Language Models
• Authors: Alireza Salemi, Hamed Zamani,
• Abstract summary: This paper presents the first systematic comparison between two approaches on a wide range of personalization tasks. We find that RAG-based and PEFT-based personalization methods on average yield 14.92% and 1.07% improvements over the non-personalized LLM, respectively.
• Score: 21.115495457454365
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Privacy-preserving methods for personalizing large language models (LLMs) are relatively under-explored. There are two schools of thought on this topic: (1) generating personalized outputs by personalizing the input prompt through retrieval augmentation from the user's personal information (RAG-based methods), and (2) parameter-efficient fine-tuning of LLMs per user that considers efficiency and space limitations (PEFT-based methods). This paper presents the first systematic comparison between two approaches on a wide range of personalization tasks using seven diverse datasets. Our results indicate that RAG-based and PEFT-based personalization methods on average yield 14.92% and 1.07% improvements over the non-personalized LLM, respectively. We find that combining RAG with PEFT elevates these improvements to 15.98%. Additionally, we identify a positive correlation between the amount of user data and PEFT's effectiveness, indicating that RAG is a better choice for cold-start users (i.e., user's with limited personal data).

Related papers

• Personalized LLM Decoding via Contrasting Personal Preference [8.469329222500726]
  We propose CoPe, a novel decoding-time approach applied after performing parameter-efficient fine-tuning (PEFT) on user-specific data.<n>Our core idea is to leverage reward-guided decoding specifically for personalization by maximizing each user's implicit reward signal.<n>Our empirical results demonstrate that CoPe achieves strong performance, improving personalization by an average of 10.57% in ROUGE-L.
  arXiv  Detail & Related papers  (2025-06-13T09:12:44Z)
• Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models [83.8639566087953]
  We propose a direct retrieval-augmented optimization framework, named DRO, that enables end-to-end training of two key components.<n>DRO alternates between two phases: (i) document permutation estimation and (ii) re-weighted, progressively improving RAG components.<n>Our theoretical analysis reveals that DRO is analogous to policy-gradient methods in reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-05T23:54:53Z)
• Personalized Language Models via Privacy-Preserving Evolutionary Model Merging [57.161917758405465]
  Personalization in large language models (LLMs) seeks to tailor models to individual user or user group preferences. We propose Privacy-Preserving Model Merging via Evolutionary Algorithms (PriME) PriME employs gradient-free methods to directly optimize task-specific metrics while preserving user privacy.
  arXiv  Detail & Related papers  (2025-03-23T09:46:07Z)
• From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment [41.96246165999026]
  Large language models (LLMs) have traditionally been aligned through one-size-fits-all approaches. This paper introduces a comprehensive framework for scalable personalized alignment of LLMs.
  arXiv  Detail & Related papers  (2025-03-19T17:41:46Z)
• Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalization [68.79814761867314]
  We propose Difference-aware Personalization Learning (DPL) to enhance Large Language Models (LLMs) personalization. DPL strategically selects representative users for comparison and establishes a structured standard to extract task-relevant differences. Experiments on real-world datasets demonstrate that DPL significantly enhances LLM personalization.
  arXiv  Detail & Related papers  (2025-03-04T09:53:26Z)
• Personalize Your LLM: Fake it then Align it [12.436528089142698]
  CHAMELEON is a scalable and efficient personalization approach that uses self-generated personal preference data and representation editing. Our experiments show that CHAMELEON efficiently adapts models to personal preferences, improving instruction-tuned models and outperforms two personalization baselines by an average of 40%.
  arXiv  Detail & Related papers  (2025-03-02T22:40:10Z)
• FSPO: Few-Shot Preference Optimization of Synthetic Preference Data in LLMs Elicits Effective Personalization to Real Users [111.56469697145519]
  We propose Few-Shot Preference Optimization, which reframes reward modeling as a meta-learning problem. Under this framework, an LLM learns to quickly adapt to a user via a few labeled preferences from that user, constructing a personalized reward function for them. We generate over 1M synthetic personalized preferences using publicly available LLMs. We evaluate FSPO on personalized open-ended generation for up to 1,500 synthetic users across three domains: movie reviews, pedagogical adaptation based on educational background, and general question answering, along with a controlled human study.
  arXiv  Detail & Related papers  (2025-02-26T17:08:46Z)
• Personalized Preference Fine-tuning of Diffusion Models [75.22218338096316]
  We introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way. Our approach achieves an average win rate of 76% over Stable Cascade, generating images that more accurately reflect specific user preferences.
  arXiv  Detail & Related papers  (2025-01-11T22:38:41Z)
• Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective [4.548047308860141]
  This study investigates the impact of different type of preference data on model performance. It aims to reduce their dependency on extensive amounts of preference data, which is expensive to collect.
  arXiv  Detail & Related papers  (2024-10-22T00:11:41Z)
• PersonalLLM: Tailoring LLMs to Individual Preferences [11.717169516971856]
  We present a public benchmark, PersonalLLM, focusing on adapting LLMs to provide maximal benefits for a particular user.<n>We curate open-ended prompts paired with many high-quality answers over which users would be expected to display heterogeneous latent preferences.<n>Our dataset and generated personalities offer an innovative testbed for developing personalization algorithms.
  arXiv  Detail & Related papers  (2024-09-30T13:55:42Z)
• Lifelong Personalized Low-Rank Adaptation of Large Language Models for Recommendation [50.837277466987345]
  We focus on the field of large language models (LLMs) for recommendation. We propose RecLoRA, which incorporates a Personalized LoRA module that maintains independent LoRAs for different users. We also design a Few2Many Learning Strategy, using a conventional recommendation model as a lens to magnify small training spaces to full spaces.
  arXiv  Detail & Related papers  (2024-08-07T04:20:28Z)
• Aligning Large Language Models with Self-generated Preference Data [72.99676237703099]
  We propose a new framework that boosts the alignment of large language models (LLMs) with human preferences. Our key idea is leveraging the human prior knowledge within the small (seed) data. We introduce a noise-aware preference learning algorithm to mitigate the risk of low quality within generated preference data.
  arXiv  Detail & Related papers  (2024-06-06T18:01:02Z)
• Adaptive Preference Scaling for Reinforcement Learning with Human Feedback [103.36048042664768]
  Reinforcement learning from human feedback (RLHF) is a prevalent approach to align AI systems with human values. We propose a novel adaptive preference loss, underpinned by distributionally robust optimization (DRO) Our method is versatile and can be readily adapted to various preference optimization frameworks.
  arXiv  Detail & Related papers  (2024-06-04T20:33:22Z)
• Efficient and Responsible Adaptation of Large Language Models for Robust Top-k Recommendations [11.004673022505566]
  Long user queries from millions of users can degrade the performance of large language models for recommendation. We propose a hybrid task allocation framework that utilizes the capabilities of both large language models and traditional recommendation systems. Our results on three real-world datasets show a significant reduction in weak users and improved robustness of RSs to sub-populations.
  arXiv  Detail & Related papers  (2024-05-01T19:11:47Z)
• Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation [56.13803674092712]
  We propose an industrial-friendly, expert-aligned and diversity-preserved instruction data selection method: Clustering and Ranking (CaR) CaR employs a two-step process: first, it ranks instruction pairs using a high-accuracy (84.25%) scoring model aligned with expert preferences; second, it preserves dataset diversity through clustering. In our experiment, CaR efficiently selected a mere 1.96% of Alpaca's IT data, yet the resulting AlpaCaR model surpassed Alpaca's performance by an average of 32.1% in GPT-4 evaluations.
  arXiv  Detail & Related papers  (2024-02-28T09:27:29Z)
• Persona-DB: Efficient Large Language Model Personalization for Response Prediction with Collaborative Data Refinement [79.2400720115588]
  We introduce Persona-DB, a simple yet effective framework consisting of a hierarchical construction process to improve generalization across task contexts. In the evaluation of response prediction, Persona-DB demonstrates superior context efficiency in maintaining accuracy with a significantly reduced retrieval size. Our experiments also indicate a marked improvement of over 10% under cold-start scenarios, when users have extremely sparse data.
  arXiv  Detail & Related papers  (2024-02-16T20:20:43Z)
• Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuning [36.88126051792774]
  Personalization in large language models (LLMs) is increasingly important. One PEFT Per User (OPPU) employs personalized parameter-efficient fine-tuning (PEFT) modules to store user-specific behavior patterns and preferences. OPPU significantly outperforms existing prompt-based methods across seven diverse tasks in the LaMP benchmark.
  arXiv  Detail & Related papers  (2024-02-06T21:03:52Z)
• Interactive Hyperparameter Optimization in Multi-Objective Problems via Preference Learning [65.51668094117802]
  We propose a human-centered interactive HPO approach tailored towards multi-objective machine learning (ML) Instead of relying on the user guessing the most suitable indicator for their needs, our approach automatically learns an appropriate indicator.
  arXiv  Detail & Related papers  (2023-09-07T09:22:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.