Quantifying Association Capabilities of Large Language Models and Its
Implications on Privacy Leakage
- URL: http://arxiv.org/abs/2305.12707v2
- Date: Fri, 9 Feb 2024 05:31:11 GMT
- Title: Quantifying Association Capabilities of Large Language Models and Its
Implications on Privacy Leakage
- Authors: Hanyin Shao, Jie Huang, Shen Zheng, Kevin Chen-Chuan Chang
- Abstract summary: This paper delves into the association capabilities of language models, aiming to uncover the factors that influence their proficiency in associating information.
Our study reveals that as models scale up, their capacity to associate entities/information intensifies, particularly when target pairs demonstrate shorter co-occurrence distances or higher co-occurrence frequencies.
Despite the proportion of accurately predicted PII being relatively small, LLMs still demonstrate the capability to predict specific instances of email addresses and phone numbers when provided with appropriate prompts.
- Score: 28.385083741414213
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The advancement of large language models (LLMs) brings notable improvements
across various applications, while simultaneously raising concerns about
potential private data exposure. One notable capability of LLMs is their
ability to form associations between different pieces of information, but this
raises concerns when it comes to personally identifiable information (PII).
This paper delves into the association capabilities of language models, aiming
to uncover the factors that influence their proficiency in associating
information. Our study reveals that as models scale up, their capacity to
associate entities/information intensifies, particularly when target pairs
demonstrate shorter co-occurrence distances or higher co-occurrence
frequencies. However, there is a distinct performance gap when associating
commonsense knowledge versus PII, with the latter showing lower accuracy.
Despite the proportion of accurately predicted PII being relatively small, LLMs
still demonstrate the capability to predict specific instances of email
addresses and phone numbers when provided with appropriate prompts. These
findings underscore the potential risk to PII confidentiality posed by the
evolving capabilities of LLMs, especially as they continue to expand in scale
and power.
Related papers
- Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data [76.90128359866462]
We investigate the interplay between generalization and memorization in large language models at scale.
With various sizes of open-source LLMs and their pretraining corpora, we observe that as the model size increases, the task-relevant $n$-gram pair data becomes increasingly important.
Our results support the hypothesis that LLMs' capabilities emerge from a delicate balance of memorization and generalization with sufficient task-related pretraining data.
arXiv Detail & Related papers (2024-07-20T21:24:40Z) - Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation [37.63939774027709]
We propose enhancing the predicted sequence probability by assigning different weights to various tokens.
We refer to this new score as the Contextualized Sequence Likelihood (CSL)
arXiv Detail & Related papers (2024-06-03T21:55:07Z) - Locally Differentially Private In-Context Learning [8.659575019965152]
Large pretrained language models (LLMs) have shown surprising In-Context Learning (ICL) ability.
This paper proposes a locally differentially private framework of in-context learning (LDP-ICL)
Considering the mechanisms of in-context learning in Transformers by gradient descent, we provide an analysis of the trade-off between privacy and utility in such LDP-ICL.
arXiv Detail & Related papers (2024-05-07T06:05:43Z) - Characterizing Truthfulness in Large Language Model Generations with
Local Intrinsic Dimension [63.330262740414646]
We study how to characterize and predict the truthfulness of texts generated from large language models (LLMs)
We suggest investigating internal activations and quantifying LLM's truthfulness using the local intrinsic dimension (LID) of model activations.
arXiv Detail & Related papers (2024-02-28T04:56:21Z) - Do Membership Inference Attacks Work on Large Language Models? [145.90022632726883]
Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data.
We perform a large-scale evaluation of MIAs over a suite of language models trained on the Pile, ranging from 160M to 12B parameters.
We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains.
arXiv Detail & Related papers (2024-02-12T17:52:05Z) - Discovery of the Hidden World with Large Language Models [100.38157787218044]
We introduce COAT: Causal representatiOn AssistanT.
COAT incorporates LLMs as a factor proposer that extracts the potential causal factors from unstructured data.
LLMs can also be instructed to provide additional information used to collect data values.
arXiv Detail & Related papers (2024-02-06T12:18:54Z) - Differentially Private Low-Rank Adaptation of Large Language Model Using Federated Learning [32.52811740662061]
This article introduces DP-LoRA, a novel federated learning algorithm tailored for large language models (LLMs)
DP-LoRA preserves data privacy by employing a Gaussian mechanism that adds noise in weight updates, maintaining individual data privacy while facilitating collaborative model training.
arXiv Detail & Related papers (2023-12-29T06:50:38Z) - Large Language Models Can Be Good Privacy Protection Learners [53.07930843882592]
We introduce Privacy Protection Language Models (PPLM), a novel paradigm for fine-tuning language models.
Our work offers a theoretical analysis for model design and delves into various techniques such as corpus curation, penalty-based unlikelihood in training loss, and instruction-based tuning.
In particular, instruction tuning with both positive and negative examples, stands out as a promising method, effectively protecting private data while enhancing the model's knowledge.
arXiv Detail & Related papers (2023-10-03T22:37:01Z) - Improving Open Information Extraction with Large Language Models: A
Study on Demonstration Uncertainty [52.72790059506241]
Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text.
Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
arXiv Detail & Related papers (2023-09-07T01:35:24Z) - Quantifying and Analyzing Entity-level Memorization in Large Language
Models [4.59914731734176]
Large language models (LLMs) have been proven capable of memorizing their training data.
Privacy risks arising from memorization have attracted increasing attention.
We propose a fine-grained, entity-level definition to quantify memorization with conditions and metrics closer to real-world scenarios.
arXiv Detail & Related papers (2023-08-30T03:06:47Z) - ProPILE: Probing Privacy Leakage in Large Language Models [38.92840523665835]
Large language models (LLMs) are often trained on vast quantities of web-collected data, which may inadvertently include sensitive personal data.
This paper presents ProPILE, a novel probing tool designed to empower data subjects, or the owners of the PII, with awareness of potential PII leakage.
arXiv Detail & Related papers (2023-07-04T18:53:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.