FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models

• URL: http://arxiv.org/abs/2406.00839v1
• Date: Sun, 2 Jun 2024 19:17:00 GMT
• Title: FOCUS: Forging Originality through Contrastive Use in Self-Plagiarism for Language Models
• Authors: Kaixin Lan, Tao Fang, Derek F. Wong, Yabo Xu, Lidia S. Chao, Cecilia G. Zhao,
• Abstract summary: Pre-trained Language Models (PLMs) have shown impressive results in various Natural Language Generation (NLG) tasks. This study introduces a unique "self-plagiarism" contrastive decoding strategy, aimed at boosting the originality of text produced by PLMs.
• Score: 38.76912842622624
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained Language Models (PLMs) have shown impressive results in various Natural Language Generation (NLG) tasks, such as powering chatbots and generating stories. However, an ethical concern arises due to their potential to produce verbatim copies of paragraphs from their training data. This is problematic as PLMs are trained on corpora constructed by human authors. As such, there is a pressing need for research to promote the generation of original content by these models. In this study, we introduce a unique "self-plagiarism" contrastive decoding strategy, aimed at boosting the originality of text produced by PLMs. Our method entails modifying prompts in LLMs to develop an amateur model and a professional model. Specifically, the amateur model is urged to plagiarize using three plagiarism templates we have designed, while the professional model maintains its standard language model status. This strategy employs prompts to stimulate the model's capacity to identify non-original candidate token combinations and subsequently impose penalties. The application of this strategy is integrated prior to the model's final layer, ensuring smooth integration with most existing PLMs (T5, GPT, LLaMA) without necessitating further adjustments. Implementing our strategy, we observe a significant decline in non-original sequences comprised of more than three words in the academic AASC dataset and the story-based ROCStories dataset.

Related papers

• A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution [57.309390098903]
  Authorship attribution aims to identify the origin or author of a document. Large Language Models (LLMs) with their deep reasoning capabilities and ability to maintain long-range textual associations offer a promising alternative. Our results on the IMDb and blog datasets show an impressive 85% accuracy in one-shot authorship classification across ten authors.
  arXiv  Detail & Related papers  (2024-10-29T04:14:23Z)
• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Detection and Measurement of Syntactic Templates in Generated Text [58.111650675717414]
  We offer an analysis of syntactic features to characterize general repetition in models. We find that models tend to produce templated text in downstream tasks at a higher rate than what is found in human-reference texts.
  arXiv  Detail & Related papers  (2024-06-28T19:34:23Z)
• PlagBench: Exploring the Duality of Large Language Models in Plagiarism Generation and Detection [26.191836276118696]
  We introduce PlagBench, a comprehensive dataset consisting of 46.5K synthetic plagiarism cases generated using three instruction-tuned LLMs. We then leverage our proposed dataset to evaluate the plagiarism detection performance of five modern LLMs and three specialized plagiarism checkers. Our findings reveal that GPT-3.5 tends to generates paraphrases and summaries of higher quality compared to Llama2 and GPT-4.
  arXiv  Detail & Related papers  (2024-06-24T03:29:53Z)
• Aligning Large Language Models with Counterfactual DPO [1.8130068086063336]
  This paper explores the utilization of counterfactual prompting to align the model's style without relying on human intervention. We demonstrate that this method effectively instils desirable behaviour, mitigates undesirable ones, and encourages the model to disregard inappropriate instructions.
  arXiv  Detail & Related papers  (2024-01-17T19:43:43Z)
• Who's Harry Potter? Approximate Unlearning in LLMs [4.821438899378393]
  Large language models (LLMs) are trained on massive internet corpora that often contain copyrighted content. This poses legal and ethical challenges for the developers and users of these models, as well as the original authors and publishers. We propose a novel technique for unlearning a subset of the training data from a LLM, without having to retrain it from scratch.
  arXiv  Detail & Related papers  (2023-10-03T17:48:14Z)
• LexGPT 0.1: pre-trained GPT-J models with Pile of Law [1.8275108630751844]
  This research aims to build generative language models specialized for the legal domain. The manuscript presents the development of LexGPT models based on GPT-J models and pre-trained with Pile of Law.
  arXiv  Detail & Related papers  (2023-06-05T08:42:59Z)
• The False Promise of Imitating Proprietary LLMs [158.65692029352584]
  An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model. This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes. We then evaluate the models using crowd raters and canonical NLP benchmarks.
  arXiv  Detail & Related papers  (2023-05-25T05:00:12Z)
• Do Language Models Plagiarize? [22.02731537718498]
  We investigate whether language models memorize but also plagiarize training samples when generating artificial texts. Our findings support that they, especially GPT-2, reuse particular pieces of texts from the training corpus with or without obfuscation. Our work implies that future research on neural language models should take precautions to avoid models plagiarizing their training datasets.
  arXiv  Detail & Related papers  (2022-03-15T03:11:11Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.