PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization

• URL: http://arxiv.org/abs/2608.29624v1
• Date: Sun, 30 Aug 2026 07:33:08 GMT
• Title: PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatization
• Abstract summary: PrivBench is a benchmarking platform for researchers and practitioners working on text privatization.<n>PrivBench is free to use and openly accessible at https://privbench.com/.
• Score: 50.692938672750266
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Natural Language Processing methods have enabled novel solutions and advances in the field of privacy, particularly in the sub-domain of text-to-text privatization, where the goal is to transform a sensitive input text into a privatized output by ideally masking (in)directly identifiable or otherwise private information. The evaluation of text-to-text privatization, however, is not straightforward, and the extant literature has utilized a myriad of techniques and metrics to quantify the privacy-preserving capabilities of privatization methods. Seeking to unify the evaluation of text-to-text privatization, we introduce PrivBench, a holistic and modular benchmarking platform for researchers and practitioners working on text privatization. PrivBench is holistic in that it evaluates privatization on a series of defined desiderata, which are structured into modules. PrivBench is not only modular but also extensible, allowing for future updates and benchmark versions. PrivBench is user-centered and promotes competition via real-time evaluation and a live public leaderboard. The platform is free to use and openly accessible at https://privbench.com/.

Related papers

• Towards Privacy-Preserving Machine Translation at the Inference Stage: A New Task and Benchmark [52.063283137654196]
  Current online translation services require sending user text to cloud servers, posing a risk of privacy leakage when the text contains sensitive information.<n>This paper proposes a novel "Privacy-Preserving Machine Translation" (PPMT) task, aiming to protect the private information in text during the model inference stage.
  arXiv  Detail & Related papers  (2026-03-16T02:41:06Z)
• Zero-Shot Privacy-Aware Text Rewriting via Iterative Tree Search [60.197239728279534]
  Large language models (LLMs) in cloud-based services have raised significant privacy concerns.<n>Existing text anonymization and de-identification techniques, such as rule-based redaction and scrubbing, often struggle to balance privacy preservation with text naturalness and utility.<n>We propose a zero-shot, tree-search-based iterative sentence rewriting algorithm that systematically obfuscates or deletes private information while preserving coherence, relevance, and naturalness.
  arXiv  Detail & Related papers  (2025-09-25T07:23:52Z)
• The Double-edged Sword of LLM-based Data Reconstruction: Understanding and Mitigating Contextual Vulnerability in Word-level Differential Privacy Text Sanitization [53.51921540246166]
  We show that Language Large Models (LLMs) can exploit the contextual vulnerability of DP-sanitized texts.<n>Experiments uncover a double-edged sword effect of LLM reconstructions on privacy and utility.<n>We propose recommendations for using data reconstruction as a post-processing step.
  arXiv  Detail & Related papers  (2025-08-26T12:22:45Z)
• Spend Your Budget Wisely: Towards an Intelligent Distribution of the Privacy Budget in Differentially Private Text Rewriting [3.0177210416625124]
  We construct and evaluate a toolkit of linguistics- and NLP-based methods used to allocate a privacy budget to constituent tokens in a text document.<n>Our work highlights the intricacies of text privatization with DP, and furthermore, it calls for further work on finding more efficient ways to maximize the privatization benefits offered by DP in text rewriting.
  arXiv  Detail & Related papers  (2025-03-28T12:33:46Z)
• Investigating User Perspectives on Differentially Private Text Privatization [81.59631769859004]
  This work investigates how factors of $textitscenario$, $textitdata sensitivity$, $textitmechanism type$, and $textitreason for data collection$ impact user preferences for text privatization.<n>We learn that while all these factors play a role in influencing privacy decisions, users are highly sensitive to the utility and coherence of the private output texts.
  arXiv  Detail & Related papers  (2025-03-12T12:33:20Z)
• Private Text Generation by Seeding Large Language Model Prompts [13.407214545457778]
  We propose Differentially Private Keyphrase Prompt Seeding (DP-KPS), a method that generates a private synthetic text corpus from a sensitive input corpus.<n>We evaluate DP-KPS on downstream ML text classification tasks, and show that the corpora it generates preserve much of the predictive power of the original ones.
  arXiv  Detail & Related papers  (2025-02-18T16:50:38Z)
• Differential Privacy Overview and Fundamental Techniques [63.0409690498569]
  This chapter is meant to be part of the book "Differential Privacy in Artificial Intelligence: From Theory to Practice" It starts by illustrating various attempts to protect data privacy, emphasizing where and why they failed. It then defines the key actors, tasks, and scopes that make up the domain of privacy-preserving data analysis.
  arXiv  Detail & Related papers  (2024-11-07T13:52:11Z)
• Privacy-Adaptive BERT for Natural Language Understanding [20.821155542969947]
  We study how to improve the effectiveness of NLU models under a Local Privacy setting using BERT. We propose privacy-adaptive LM pretraining methods and demonstrate that they can significantly improve model performance on privatized text input.
  arXiv  Detail & Related papers  (2021-04-15T15:01:28Z)
• PGLP: Customizable and Rigorous Location Privacy through Policy Graph [68.3736286350014]
  We propose a new location privacy notion called PGLP, which provides a rich interface to release private locations with customizable and rigorous privacy guarantee. Specifically, we formalize a user's location privacy requirements using a textitlocation policy graph, which is expressive and customizable. Third, we design a private location trace release framework that pipelines the detection of location exposure, policy graph repair, and private trajectory release with customizable and rigorous location privacy.
  arXiv  Detail & Related papers  (2020-05-04T04:25:59Z)
• PrivEdge: From Local to Distributed Private Training and Prediction [43.02041269239928]
  PrivEdge is a technique for privacy-preserving Machine Learning (ML) PrivEdge safeguards the privacy of users who provide their data for training, as well as users who use the prediction service. We show that PrivEdge has high precision and recall in preserving privacy, as well as in distinguishing between private and non-private images.
  arXiv  Detail & Related papers  (2020-04-12T09:26:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.