Sentence-level Privacy for Document Embeddings

• URL: http://arxiv.org/abs/2205.04605v1
• Date: Tue, 10 May 2022 00:19:35 GMT
• Title: Sentence-level Privacy for Document Embeddings
• Authors: Casey Meehan, Khalil Mrini, Kamalika Chaudhuri
• Abstract summary: We propose SentDP: pure local differential privacy at the sentence level for a single user document. Our experiments indicate that these private document embeddings are useful for downstream tasks like sentiment analysis and topic classification.
• Score: 25.779351166096255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: User language data can contain highly sensitive personal content. As such, it is imperative to offer users a strong and interpretable privacy guarantee when learning from their data. In this work, we propose SentDP: pure local differential privacy at the sentence level for a single user document. We propose a novel technique, DeepCandidate, that combines concepts from robust statistics and language modeling to produce high-dimensional, general-purpose $\epsilon$-SentDP document embeddings. This guarantees that any single sentence in a document can be substituted with any other sentence while keeping the embedding $\epsilon$-indistinguishable. Our experiments indicate that these private document embeddings are useful for downstream tasks like sentiment analysis and topic classification and even outperform baseline methods with weaker guarantees like word-level Metric DP.

Related papers

• Spend Your Budget Wisely: Towards an Intelligent Distribution of the Privacy Budget in Differentially Private Text Rewriting [3.0177210416625124]
  We construct and evaluate a toolkit of linguistics- and NLP-based methods used to allocate a privacy budget to constituent tokens in a text document. Our work highlights the intricacies of text privatization with DP, and furthermore, it calls for further work on finding more efficient ways to maximize the privatization benefits offered by DP in text rewriting.
  arXiv  Detail & Related papers  (2025-03-28T12:33:46Z)
• Contextual Document Embeddings [77.22328616983417]
  We propose two complementary methods for contextualized document embeddings. First, an alternative contrastive learning objective that explicitly incorporates the document neighbors into the intra-batch contextual loss. Second, a new contextual architecture that explicitly encodes neighbor document information into the encoded representation.
  arXiv  Detail & Related papers  (2024-10-03T14:33:34Z)
• A Collocation-based Method for Addressing Challenges in Word-level Metric Differential Privacy [3.0177210416625124]
  Several word-level $textitMetric$ Differential Privacy approaches have been proposed. We devise a method where composed privatized outputs have higher semantic coherence and variable length. We evaluate our method in utility and privacy tests, which make a clear case for tokenization strategies beyond the word level.
  arXiv  Detail & Related papers  (2024-06-30T09:37:34Z)
• Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text [3.3916160303055567]
  We propose a simple post-processing method based on the goal of aligning rewritten texts with their original counterparts. Our results show that such an approach not only produces outputs that are more semantically reminiscent of the original inputs, but also texts which score on average better in empirical privacy evaluations.
  arXiv  Detail & Related papers  (2024-05-30T08:41:33Z)
• Metric Differential Privacy at the User-Level Via the Earth Mover's Distance [34.63551774740707]
  Metric differential privacy (DP) provides heterogeneous privacy guarantees based on a distance between the pair of inputs. In this paper, we initiate the study of one natural definition of metric DP at the user-level. We design two novel mechanisms under $d_textsfEM$-DP to answer linear queries and item-wise queries.
  arXiv  Detail & Related papers  (2024-05-04T13:29:11Z)
• Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training [55.321010757641524]
  A major public concern regarding the training of large language models (LLMs) is whether they abusing copyrighted online text. Previous membership inference methods may be misled by similar examples in vast amounts of training data. We propose an alternative textitinsert-and-detection methodology, advocating that web users and content platforms employ textbftextitunique identifiers.
  arXiv  Detail & Related papers  (2024-03-23T06:36:32Z)
• Unsupervised Sentiment Analysis of Plastic Surgery Social Media Posts [91.3755431537592]
  The massive collection of user posts across social media platforms is primarily untapped for artificial intelligence (AI) use cases. Natural language processing (NLP) is a subfield of AI that leverages bodies of documents, known as corpora, to train computers in human-like language understanding. This study demonstrates that the applied results of unsupervised analysis allow a computer to predict either negative, positive, or neutral user sentiment towards plastic surgery.
  arXiv  Detail & Related papers  (2023-07-05T20:16:20Z)
• Are the Best Multilingual Document Embeddings simply Based on Sentence Embeddings? [18.968571816913208]
  We provide a systematic comparison of methods to produce document-level representations from sentences based on LASER, LaBSE, and Sentence BERT pre-trained multilingual models. We show that a clever combination of sentence embeddings is usually better than encoding the full document as a single unit.
  arXiv  Detail & Related papers  (2023-04-28T12:11:21Z)
• DP-BART for Privatized Text Rewriting under Local Differential Privacy [2.45626162429986]
  We propose a new system 'DP-BART' that largely outperforms existing LDP systems. Our approach uses a novel clipping method, iterative pruning, and further training of internal representations which drastically reduces the amount of noise required for DP guarantees.
  arXiv  Detail & Related papers  (2023-02-15T13:07:34Z)
• DocNLI: A Large-scale Dataset for Document-level Natural Language Inference [55.868482696821815]
  Natural language inference (NLI) is formulated as a unified framework for solving various NLP problems. This work presents DocNLI -- a newly-constructed large-scale dataset for document-level NLI.
  arXiv  Detail & Related papers  (2021-06-17T13:02:26Z)
• Intrinsic Probing through Dimension Selection [69.52439198455438]
  Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be possible unless some form of linguistic structure inheres in these representations, and a wealth of research has sprung up on probing for it. In this paper, we draw a distinction between intrinsic probing, which examines how linguistic information is structured within a representation, and the extrinsic probing popular in prior work, which only argues for the presence of such information by showing that it can be successfully extracted.
  arXiv  Detail & Related papers  (2020-10-06T15:21:08Z)
• Learning to Select Bi-Aspect Information for Document-Scale Text Content Manipulation [50.01708049531156]
  We focus on a new practical task, document-scale text content manipulation, which is the opposite of text style transfer. In detail, the input is a set of structured records and a reference text for describing another recordset. The output is a summary that accurately describes the partial content in the source recordset with the same writing style of the reference.
  arXiv  Detail & Related papers  (2020-02-24T12:52:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.