Q-LSTM Language Model -- Decentralized Quantum Multilingual Pre-Trained Language Model for Privacy Protection

• URL: http://arxiv.org/abs/2210.03221v1
• Date: Thu, 6 Oct 2022 21:29:17 GMT
• Title: Q-LSTM Language Model -- Decentralized Quantum Multilingual Pre-Trained Language Model for Privacy Protection
• Authors: Shuyue Stella Li, Xiangyu Zhang, Shu Zhou, Hongchao Shu, Ruixing Liang, Hexin Liu, and Leibny Paola Garcia
• Abstract summary: Large-scale language models are trained on a massive amount of natural language data that might encode or reflect our private information. malicious agents can reverse engineer the training data even if data sanitation and differential privacy algorithms were involved in the pre-training process. We propose a decentralized training framework to address privacy concerns in training large-scale language models.
• Score: 6.0038761646405225
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large-scale language models are trained on a massive amount of natural language data that might encode or reflect our private information. With careful manipulation, malicious agents can reverse engineer the training data even if data sanitation and differential privacy algorithms were involved in the pre-training process. In this work, we propose a decentralized training framework to address privacy concerns in training large-scale language models. The framework consists of a cloud quantum language model built with Variational Quantum Classifiers (VQC) for sentence embedding and a local Long-Short Term Memory (LSTM) model. We use both intrinsic evaluation (loss, perplexity) and extrinsic evaluation (downstream sentiment analysis task) to evaluate the performance of our quantum language model. Our quantum model was comparable to its classical counterpart on all the above metrics. We also perform ablation studies to look into the effect of the size of VQC and the size of training data on the performance of the model. Our approach solves privacy concerns without sacrificing downstream task performance. The intractability of quantum operations on classical hardware ensures the confidentiality of the training data and makes it impossible to be recovered by any adversary.

Related papers

• Prospects of Privacy Advantage in Quantum Machine Learning [3.7592122147132767]
  This study is motivated by the increasing success of recovering input data from the gradients of classical models. We uncover the crucial role played by the Lie algebra (DLA) of the VQC ansatz in determining privacy vulnerabilities.
  arXiv  Detail & Related papers  (2024-05-14T17:49:18Z)
• PrivacyMind: Large Language Models Can Be Contextual Privacy Protection Learners [81.571305826793]
  We introduce Contextual Privacy Protection Language Models (PrivacyMind) Our work offers a theoretical analysis for model design and benchmarks various techniques. In particular, instruction tuning with both positive and negative examples stands out as a promising method.
  arXiv  Detail & Related papers  (2023-10-03T22:37:01Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Differentially Private Decoding in Large Language Models [14.221692239892207]
  We propose a simple, easy to interpret, and computationally lightweight perturbation mechanism to be applied to an already trained model at the decoding stage. Our perturbation mechanism is model-agnostic and can be used in conjunction with any Large Language Model.
  arXiv  Detail & Related papers  (2022-05-26T20:50:58Z)
• Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density [58.64907136562178]
  We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods. We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments. The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
  arXiv  Detail & Related papers  (2021-11-02T15:48:28Z)
• Quantum machine learning with differential privacy [3.2442879131520126]
  We develop a hybrid quantum-classical model that is trained to preserve privacy using differentially private optimization algorithm. Experiments demonstrate that differentially private QML can protect user-sensitive information without diminishing model accuracy.
  arXiv  Detail & Related papers  (2021-03-10T18:06:15Z)
• Training Data Leakage Analysis in Language Models [6.843491191969066]
  We introduce a methodology that investigates identifying the user content in the training data that could be leaked under a strong and realistic threat model. We propose two metrics to quantify user-level data leakage by measuring a model's ability to produce unique sentence fragments within training data.
  arXiv  Detail & Related papers  (2021-01-14T00:57:32Z)
• Pre-Training a Language Model Without Human Language [74.11825654535895]
  We study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance. We find that models pre-trained on unstructured data beat those trained directly from scratch on downstream tasks. To our great astonishment, we uncover that pre-training on certain non-human language data gives GLUE performance close to performance pre-trained on another non-English language.
  arXiv  Detail & Related papers  (2020-12-22T13:38:06Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Differentially Private Language Models Benefit from Public Pre-training [1.2676356746752895]
  We study the feasibility of learning a language model which is simultaneously high-quality and privacy preserving. We find that DP fine-tuning boosts the performance of language models in the private domain.
  arXiv  Detail & Related papers  (2020-09-13T00:50:44Z)
• How Context Affects Language Models' Factual Predictions [134.29166998377187]
  We integrate information from a retrieval system with a pre-trained language model in a purely unsupervised way. We report that augmenting pre-trained language models in this way dramatically improves performance and that the resulting system, despite being unsupervised, is competitive with a supervised machine reading baseline.
  arXiv  Detail & Related papers  (2020-05-10T09:28:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.