Make Text Unlearnable: Exploiting Effective Patterns to Protect Personal Data

• URL: http://arxiv.org/abs/2307.00456v1
• Date: Sun, 2 Jul 2023 02:34:57 GMT
• Title: Make Text Unlearnable: Exploiting Effective Patterns to Protect Personal Data
• Authors: Xinzhe Li, Ming Liu, Shang Gao
• Abstract summary: This paper addresses the ethical concerns arising from the use of unauthorized public data in deep learning models. We extend Huang et al.'s bi-level optimization approach to generate unlearnable text using a gradient-based search technique. We extract simple patterns from unlearnable text produced by bi-level optimization and demonstrate that the data remains unlearnable for unknown models.
• Score: 9.380410177526425
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the ethical concerns arising from the use of unauthorized public data in deep learning models and proposes a novel solution. Specifically, building on the work of Huang et al. (2021), we extend their bi-level optimization approach to generate unlearnable text using a gradient-based search technique. However, although effective, this approach faces practical limitations, including the requirement of batches of instances and model architecture knowledge that is not readily accessible to ordinary users with limited access to their own data. Furthermore, even with semantic-preserving constraints, unlearnable noise can alter the text's semantics. To address these challenges, we extract simple patterns from unlearnable text produced by bi-level optimization and demonstrate that the data remains unlearnable for unknown models. Additionally, these patterns are not instance- or dataset-specific, allowing users to readily apply them to text classification and question-answering tasks, even if only a small proportion of users implement them on their public content. We also open-source codes to generate unlearnable text and assess unlearnable noise to benefit the public and future studies.

Related papers

• Towards Operationalizing Right to Data Protection [8.61230665736263]
  RegText is a framework that injects imperceptible correlations into natural language datasets effectively rendering them unlearnable without affecting content. We demonstrate RegText's utility through rigorous empirical analysis of small and large LMs. RegText can newer models like GPT-4o and Llama from learning on our generated data, resulting in a drop in their test accuracy compared to their zero-shot performance.
  arXiv  Detail & Related papers  (2024-11-13T10:43:31Z)
• MUSE: Machine Unlearning Six-Way Evaluation for Language Models [109.76505405962783]
  Language models (LMs) are trained on vast amounts of text data, which may include private and copyrighted content. We propose MUSE, a comprehensive machine unlearning evaluation benchmark. We benchmark how effectively eight popular unlearning algorithms can unlearn Harry Potter books and news articles.
  arXiv  Detail & Related papers  (2024-07-08T23:47:29Z)
• IDT: Dual-Task Adversarial Attacks for Privacy Protection [8.312362092693377]
  Methods to protect privacy can involve using representations inside models that are not to detect sensitive attributes. We propose IDT, a method that analyses predictions made by auxiliary and interpretable models to identify which tokens are important to change. We evaluate different datasets for NLP suitable for different tasks.
  arXiv  Detail & Related papers  (2024-06-28T04:14:35Z)
• Machine Unlearning in Large Language Models [0.7864304771129751]
  This paper introduces a methodology to align large language models (LLMs) with ethical, privacy, and safety standards. Our approach aims to selectively erase or modify learned information in LLMs, targeting harmful responses and copyrighted content.
  arXiv  Detail & Related papers  (2024-05-24T02:12:51Z)
• Maximizing Data Efficiency for Cross-Lingual TTS Adaptation by Self-Supervised Representation Mixing and Embedding Initialization [57.38123229553157]
  This paper presents an effective transfer learning framework for language adaptation in text-to-speech systems. We focus on achieving language adaptation using minimal labeled and unlabeled data. Experimental results show that our framework is able to synthesize intelligible speech in unseen languages with only 4 utterances of labeled data and 15 minutes of unlabeled data.
  arXiv  Detail & Related papers  (2024-01-23T21:55:34Z)
• Privacy Leakage in Text Classification: A Data Extraction Approach [9.045332526072828]
  We study the potential privacy leakage in the text classification domain by investigating the problem of unintended memorization of training data. We propose an algorithm to extract missing tokens of a partial text by exploiting the likelihood of the class label provided by the model.
  arXiv  Detail & Related papers  (2022-06-09T16:14:26Z)
• Annotation Error Detection: Analyzing the Past and Present for a More Coherent Future [63.99570204416711]
  We reimplement 18 methods for detecting potential annotation errors and evaluate them on 9 English datasets. We define a uniform evaluation setup including a new formalization of the annotation error detection task. We release our datasets and implementations in an easy-to-use and open source software package.
  arXiv  Detail & Related papers  (2022-06-05T22:31:45Z)
• Toward the Understanding of Deep Text Matching Models for Information Retrieval [72.72380690535766]
  This paper aims at testing whether existing deep text matching methods satisfy some fundamental gradients in information retrieval. Specifically, four attributions are used in our study, i.e., term frequency constraint, term discrimination constraint, length normalization constraints, and TF-length constraint. Experimental results on LETOR 4.0 and MS Marco show that all the investigated deep text matching methods satisfy the above constraints with high probabilities in statistics.
  arXiv  Detail & Related papers  (2021-08-16T13:33:15Z)
• A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation [50.55448707570669]
  We propose a novel token-level, reference-free hallucination detection task and an associated annotated dataset named HaDes. To create this dataset, we first perturb a large number of text segments extracted from English language Wikipedia, and then verify these with crowd-sourced annotations.
  arXiv  Detail & Related papers  (2021-04-18T04:09:48Z)
• Self-Supervised Noisy Label Learning for Source-Free Unsupervised Domain Adaptation [87.60688582088194]
  We propose a novel Self-Supervised Noisy Label Learning method. Our method can easily achieve state-of-the-art results and surpass other methods by a very large margin.
  arXiv  Detail & Related papers  (2021-02-23T10:51:45Z)
• Controlled Hallucinations: Learning to Generate Faithfully from Noisy Data [1.0914300987810126]
  We present a technique to treat such hallucinations as a controllable aspect of the generated text. On the WikiBio corpus, a particularly noisy dataset, we demonstrate the efficacy of the technique both in an automatic and in a human evaluation.
  arXiv  Detail & Related papers  (2020-10-12T17:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.