DE-COP: Detecting Copyrighted Content in Language Models Training Data

• URL: http://arxiv.org/abs/2402.09910v2
• Date: Tue, 25 Jun 2024 10:33:41 GMT
• Title: DE-COP: Detecting Copyrighted Content in Language Models Training Data
• Authors: André V. Duarte, Xuandong Zhao, Arlindo L. Oliveira, Lei Li,
• Abstract summary: We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff. Experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance.
• Score: 24.15936677068714
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How can we detect if copyrighted content was used in the training process of a language model, considering that the training data is typically undisclosed? We are motivated by the premise that a language model is likely to identify verbatim excerpts from its training text. We propose DE-COP, a method to determine whether a piece of copyrighted content was included in training. DE-COP's core approach is to probe an LLM with multiple-choice questions, whose options include both verbatim text and their paraphrases. We construct BookTection, a benchmark with excerpts from 165 books published prior and subsequent to a model's training cutoff, along with their paraphrases. Our experiments show that DE-COP surpasses the prior best method by 9.6% in detection performance (AUC) on models with logits available. Moreover, DE-COP also achieves an average accuracy of 72% for detecting suspect books on fully black-box models where prior methods give approximately 4% accuracy. The code and datasets are available at https://github.com/LeiLiLab/DE-COP.

Related papers

• A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution [57.309390098903]
  Authorship attribution aims to identify the origin or author of a document. Large Language Models (LLMs) with their deep reasoning capabilities and ability to maintain long-range textual associations offer a promising alternative. Our results on the IMDb and blog datasets show an impressive 85% accuracy in one-shot authorship classification across ten authors.
  arXiv  Detail & Related papers  (2024-10-29T04:14:23Z)
• Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method [108.56493934296687]
  We introduce a divergence-based calibration method, inspired by the divergence-from-randomness concept, to calibrate token probabilities for pretraining data detection. We have developed a Chinese-language benchmark, PatentMIA, to assess the performance of detection approaches for LLMs on Chinese text.
  arXiv  Detail & Related papers  (2024-09-23T07:55:35Z)
• Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training [55.321010757641524]
  A major public concern regarding the training of large language models (LLMs) is whether they abusing copyrighted online text. Previous membership inference methods may be misled by similar examples in vast amounts of training data. We propose an alternative textitinsert-and-detection methodology, advocating that web users and content platforms employ textbftextitunique identifiers.
  arXiv  Detail & Related papers  (2024-03-23T06:36:32Z)
• Copyright Traps for Large Language Models [6.902279764206365]
  We propose to use copyright traps to detect the use of copyrighted content in large language models. We train a 1.3B model from scratch and insert traps into original content (books) We show, contrary to intuition, that even medium-length trap sentences repeated a significant number of times (100) are not detectable using existing methods.
  arXiv  Detail & Related papers  (2024-02-14T18:09:53Z)
• Detecting Pretraining Data from Large Language Models [90.12037980837738]
  We study the pretraining data detection problem. Given a piece of text and black-box access to an LLM without knowing the pretraining data, can we determine if the model was trained on the provided text? We introduce a new detection method Min-K% Prob based on a simple hypothesis.
  arXiv  Detail & Related papers  (2023-10-25T17:21:23Z)
• Turning a CLIP Model into a Scene Text Detector [56.86413150091367]
  Recently, pretraining approaches based on vision language models have made effective progresses in the field of text detection. This paper proposes a new method, termed TCM, focusing on Turning the CLIP Model directly for text detection without pretraining process.
  arXiv  Detail & Related papers  (2023-02-28T06:06:12Z)
• PART: Pre-trained Authorship Representation Transformer [64.78260098263489]
  Authors writing documents imprint identifying information within their texts: vocabulary, registry, punctuation, misspellings, or even emoji usage. Previous works use hand-crafted features or classification tasks to train their authorship models, leading to poor performance on out-of-domain authors. We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)
• Multitask Training with Text Data for End-to-End Speech Recognition [45.35605825009208]
  We propose a multitask training method for attention-based end-to-end speech recognition models. We regularize the decoder in a listen, attend, and spell model by multitask training it on both audio-text and text-only data.
  arXiv  Detail & Related papers  (2020-10-27T14:29:28Z)
• Bootstrapping Weakly Supervised Segmentation-free Word Spotting through HMM-based Alignment [0.5076419064097732]
  We propose an approach that utilises transcripts without bounding box annotations to train word spotting models. This is done through a training-free alignment procedure based on hidden Markov models. We believe that this will be a significant advance towards a more general use of word spotting, since digital transcription data will already exist for parts of many collections of interest.
  arXiv  Detail & Related papers  (2020-03-24T19:41:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.