FedJudge: Federated Legal Large Language Model

• URL: http://arxiv.org/abs/2309.08173v3
• Date: Wed, 10 Apr 2024 13:24:55 GMT
• Title: FedJudge: Federated Legal Large Language Model
• Authors: Linan Yue, Qi Liu, Yichao Du, Weibo Gao, Ye Liu, Fangzhou Yao,
• Abstract summary: Large Language Models (LLMs) have gained prominence in the field of Legal Intelligence, offering potential applications in assisting legal professionals and laymen. This paper explores the integration of Legal LLMs with Federated Learning (FL) methodologies. We propose the first Federated Legal Large Language Model (FedJudge) framework, which fine-tunes Legal LLMs efficiently and effectively.
• Score: 10.70953602515144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have gained prominence in the field of Legal Intelligence, offering potential applications in assisting legal professionals and laymen. However, the centralized training of these Legal LLMs raises data privacy concerns, as legal data is distributed among various institutions containing sensitive individual information. This paper addresses this challenge by exploring the integration of Legal LLMs with Federated Learning (FL) methodologies. By employing FL, Legal LLMs can be fine-tuned locally on devices or clients, and their parameters are aggregated and distributed on a central server, ensuring data privacy without directly sharing raw data. However, computation and communication overheads hinder the full fine-tuning of LLMs under the FL setting. Moreover, the distribution shift of legal data reduces the effectiveness of FL methods. To this end, in this paper, we propose the first Federated Legal Large Language Model (FedJudge) framework, which fine-tunes Legal LLMs efficiently and effectively. Specifically, FedJudge utilizes parameter-efficient fine-tuning methods to update only a few additional parameters during the FL training. Besides, we explore the continual learning methods to preserve the global model's important parameters when training local clients to mitigate the problem of data shifts. Extensive experimental results on three real-world datasets clearly validate the effectiveness of FedJudge. Code is released at https://github.com/yuelinan/FedJudge.

Related papers

• CELLM: An Efficient Communication in Large Language Models Training for Federated Learning [0.0]
  This thesis aims to develop efficient training methods for large language models (LLMs) in Federated Learning (FL) First, we use low-rank adaptation (LoRA) to reduce the computational load of local model training. Second, we communicate sparse updates throughout training to significantly cut down on communication costs.
  arXiv  Detail & Related papers  (2024-07-30T05:24:08Z)
• FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model [48.33280660752336]
  Large language models (LLMs) show amazing performance on many domain-specific tasks after fine-tuning with some appropriate data. Many domain-specific data are privately distributed across multiple owners. We introduce FedBiOT, a resource-efficient LLM fine-tuning approach to federated learning.
  arXiv  Detail & Related papers  (2024-06-25T16:45:47Z)
• FedMAP: Unlocking Potential in Personalized Federated Learning through Bi-Level MAP Optimization [11.040916982022978]
  Federated Learning (FL) enables collaborative training of machine learning models on decentralized data. Data across clients often differs significantly due to class imbalance, feature distribution skew, sample size imbalance, and other phenomena. We propose a novel Bayesian PFL framework using bi-level optimization to tackle the data heterogeneity challenges.
  arXiv  Detail & Related papers  (2024-05-29T11:28:06Z)
• Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning [32.52811740662061]
  This article introduces DP-LoRA, a novel federated learning algorithm tailored for large language models (LLMs) DP-LoRA preserves data privacy by employing a Gaussian mechanism that adds noise in weight updates, maintaining individual data privacy while facilitating collaborative model training.
  arXiv  Detail & Related papers  (2023-12-29T06:50:38Z)
• Tunable Soft Prompts are Messengers in Federated Learning [55.924749085481544]
  Federated learning (FL) enables multiple participants to collaboratively train machine learning models using decentralized data sources. The lack of model privacy protection in FL becomes an unneglectable challenge. We propose a novel FL training approach that accomplishes information exchange among participants via tunable soft prompts.
  arXiv  Detail & Related papers  (2023-11-12T11:01:10Z)
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore [159.21914121143885]
  We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text. Access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile.
  arXiv  Detail & Related papers  (2023-08-08T17:58:15Z)
• FedFOR: Stateless Heterogeneous Federated Learning with First-Order Regularization [24.32029125031383]
  Federated Learning (FL) seeks to distribute model training across local clients without collecting data in a centralized data-center. We propose a first-order approximation of the global data distribution into local objectives, which intuitively penalizes updates in the opposite direction of the global update. Our approach does not impose unrealistic limits on the client size, enabling learning from a large number of clients as is typical in most FL applications.
  arXiv  Detail & Related papers  (2022-09-21T17:57:20Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)
• Do Gradient Inversion Attacks Make Federated Learning Unsafe? [70.0231254112197]
  Federated learning (FL) allows the collaborative training of AI models without needing to share raw data. Recent works on the inversion of deep neural networks from model gradients raised concerns about the security of FL in preventing the leakage of training data. In this work, we show that these attacks presented in the literature are impractical in real FL use-cases and provide a new baseline attack.
  arXiv  Detail & Related papers  (2022-02-14T18:33:12Z)
• Local Learning Matters: Rethinking Data Heterogeneity in Federated Learning [61.488646649045215]
  Federated learning (FL) is a promising strategy for performing privacy-preserving, distributed learning with a network of clients (i.e., edge devices)
  arXiv  Detail & Related papers  (2021-11-28T19:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.