Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models

• URL: http://arxiv.org/abs/2501.13904v3
• Date: Thu, 13 Feb 2025 18:58:14 GMT
• Title: Privacy-Preserving Personalized Federated Prompt Learning for Multimodal Large Language Models
• Authors: Linh Tran, Wei Sun, Stacy Patterson, Ana Milanova,
• Abstract summary: We propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle the challenge of balancing personalization and generalization. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization.
• Score: 11.747329476179223
• License:
• Abstract: Multimodal Large Language Models (LLMs) are pivotal in revolutionizing customer support and operations by integrating multiple modalities such as text, images, and audio. Federated Prompt Learning (FPL) is a recently proposed approach that combines pre-trained multimodal LLMs such as vision-language models with federated learning to create personalized, privacy-preserving AI systems. However, balancing the competing goals of personalization, generalization, and privacy remains a significant challenge. Over-personalization can lead to overfitting, reducing generalizability, while stringent privacy measures, such as differential privacy, can hinder both personalization and generalization. In this paper, we propose a Differentially Private Federated Prompt Learning (DP-FPL) approach to tackle this challenge by leveraging a low-rank factorization scheme to capture generalization while maintaining a residual term that preserves expressiveness for personalization. To ensure privacy, we introduce a novel method where we apply local differential privacy to the two low-rank components of the local prompt, and global differential privacy to the global prompt. Our approach mitigates the impact of privacy noise on the model performance while balancing the tradeoff between personalization and generalization. Extensive experiments demonstrate the effectiveness of our approach over other benchmarks.

Related papers

• Privacy in Fine-tuning Large Language Models: Attacks, Defenses, and Future Directions [11.338466798715906]
  Fine-tuning Large Language Models (LLMs) can achieve state-of-the-art performance across various domains. This paper provides a comprehensive survey of privacy challenges associated with fine-tuning LLMs. We highlight vulnerabilities to various privacy attacks, including membership inference, data extraction, and backdoor attacks.
  arXiv  Detail & Related papers  (2024-12-21T06:41:29Z)
• Few-shot Steerable Alignment: Adapting Rewards and LLM Policies with Neural Processes [50.544186914115045]
  Large language models (LLMs) are increasingly embedded in everyday applications. Ensuring their alignment with the diverse preferences of individual users has become a critical challenge. We present a novel framework for few-shot steerable alignment.
  arXiv  Detail & Related papers  (2024-12-18T16:14:59Z)
• Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions [0.0]
  This survey explores the landscape of privacy-preserving mechanisms tailored for large language models. We examine their efficacy in addressing key privacy challenges, such as membership inference and model inversion attacks. By synthesizing state-of-the-art approaches and future trends, this paper provides a foundation for developing robust, privacy-preserving large language models.
  arXiv  Detail & Related papers  (2024-12-09T00:24:09Z)
• Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuning [62.224804688233]
  differential privacy (DP) offers a promising solution by ensuring models are 'almost indistinguishable' with or without any particular privacy unit. We study user-level DP motivated by applications where it necessary to ensure uniform privacy protection across users.
  arXiv  Detail & Related papers  (2024-06-20T13:54:32Z)
• HRNet: Differentially Private Hierarchical and Multi-Resolution Network for Human Mobility Data Synthesization [19.017342515321918]
  We introduce the Hierarchical and Multi-Resolution Network (HRNet), a novel deep generative model designed to synthesize realistic human mobility data. We first identify the key difficulties inherent in learning human mobility data under differential privacy. HRNet integrates three components: a hierarchical location encoding mechanism, multi-task learning across multiple resolutions, and private pre-training.
  arXiv  Detail & Related papers  (2024-05-13T12:56:24Z)
• Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond [87.1712108247199]
  Our goal is to establish a Unified paradigm for Multi-modal Personalization systems (UniMP) We develop a generic and personalization generative framework, that can handle a wide range of personalized needs. Our methodology enhances the capabilities of foundational language models for personalized tasks.
  arXiv  Detail & Related papers  (2024-03-15T20:21:31Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Safeguarding Data in Multimodal AI: A Differentially Private Approach to CLIP Training [15.928338716118697]
  We introduce a differentially private adaptation of the Contrastive Language-Image Pretraining (CLIP) model. Our proposed method, Dp-CLIP, is rigorously evaluated on benchmark datasets.
  arXiv  Detail & Related papers  (2023-06-13T23:32:09Z)
• FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations [53.268801169075836]
  We propose FedLAP-DP, a novel privacy-preserving approach for federated learning. A formal privacy analysis demonstrates that FedLAP-DP incurs the same privacy costs as typical gradient-sharing schemes. Our approach presents a faster convergence speed compared to typical gradient-sharing methods.
  arXiv  Detail & Related papers  (2023-02-02T12:56:46Z)
• Private Reinforcement Learning with PAC and Regret Guarantees [69.4202374491817]
  We design privacy preserving exploration policies for episodic reinforcement learning (RL) We first provide a meaningful privacy formulation using the notion of joint differential privacy (JDP) We then develop a private optimism-based learning algorithm that simultaneously achieves strong PAC and regret bounds, and enjoys a JDP guarantee.
  arXiv  Detail & Related papers  (2020-09-18T20:18:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.