A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications

• URL: http://arxiv.org/abs/2202.02013v1
• Date: Fri, 4 Feb 2022 08:16:56 GMT
• Title: A Benchmark Corpus for the Detection of Automatically Generated Text in Academic Publications
• Authors: Vijini Liyanage, Davide Buscaldi, Adeline Nazarenko
• Abstract summary: This paper presents two datasets comprised of artificially generated research content. In the first case, the content is completely generated by the GPT-2 model after a short prompt extracted from original papers. The partial or hybrid dataset is created by replacing several sentences of abstracts with sentences that are generated by the Arxiv-NLP model. We evaluate the quality of the datasets comparing the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE.
• Score: 0.02578242050187029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic text generation based on neural language models has achieved performance levels that make the generated text almost indistinguishable from those written by humans. Despite the value that text generation can have in various applications, it can also be employed for malicious tasks. The diffusion of such practices represent a threat to the quality of academic publishing. To address these problems, we propose in this paper two datasets comprised of artificially generated research content: a completely synthetic dataset and a partial text substitution dataset. In the first case, the content is completely generated by the GPT-2 model after a short prompt extracted from original papers. The partial or hybrid dataset is created by replacing several sentences of abstracts with sentences that are generated by the Arxiv-NLP model. We evaluate the quality of the datasets comparing the generated texts to aligned original texts using fluency metrics such as BLEU and ROUGE. The more natural the artificial texts seem, the more difficult they are to detect and the better is the benchmark. We also evaluate the difficulty of the task of distinguishing original from generated text by using state-of-the-art classification models.

Related papers

• Detecting Document-level Paraphrased Machine Generated Content: Mimicking Human Writing Style and Involving Discourse Features [57.34477506004105]
  Machine-generated content poses challenges such as academic plagiarism and the spread of misinformation. We introduce novel methodologies and datasets to overcome these challenges. We propose MhBART, an encoder-decoder model designed to emulate human writing style. We also propose DTransformer, a model that integrates discourse analysis through PDTB preprocessing to encode structural features.
  arXiv  Detail & Related papers  (2024-12-17T08:47:41Z)
• QAEA-DR: A Unified Text Augmentation Framework for Dense Retrieval [12.225881591629815]
  In dense retrieval, embedding long texts into dense vectors can result in information loss, leading to inaccurate query-text matching. Recent studies mainly focus on improving the sentence embedding model or retrieval process. We introduce a novel text augmentation framework for dense retrieval, which transforms raw documents into information-dense text formats.
  arXiv  Detail & Related papers  (2024-07-29T17:39:08Z)
• Text2Data: Low-Resource Data Generation with Textual Control [100.5970757736845]
  Text2Data is a novel approach that utilizes unlabeled data to understand the underlying data distribution. It undergoes finetuning via a novel constraint optimization-based learning objective that ensures controllability and effectively counteracts catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-02-08T03:41:39Z)
• Enhancing Scene Text Detectors with Realistic Text Image Synthesis Using Diffusion Models [63.99110667987318]
  We present DiffText, a pipeline that seamlessly blends foreground text with the background's intrinsic features. With fewer text instances, our produced text images consistently surpass other synthetic data in aiding text detectors.
  arXiv  Detail & Related papers  (2023-11-28T06:51:28Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)
• How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN [63.79300884115027]
  Current language models can generate high-quality text. Are they simply copying text they have seen before, or have they learned generalizable linguistic abstractions? We introduce RAVEN, a suite of analyses for assessing the novelty of generated text.
  arXiv  Detail & Related papers  (2021-11-18T04:07:09Z)
• One-shot Compositional Data Generation for Low Resource Handwritten Text Recognition [10.473427493876422]
  Low resource Handwritten Text Recognition is a hard problem due to the scarce annotated data and the very limited linguistic information. In this paper we address this problem through a data generation technique based on Bayesian Program Learning. Contrary to traditional generation approaches, which require a huge amount of annotated images, our method is able to generate human-like handwriting using only one sample of each symbol from the desired alphabet.
  arXiv  Detail & Related papers  (2021-05-11T18:53:01Z)
• Neural Data-to-Text Generation with LM-based Text Augmentation [27.822282190362856]
  We show that a weakly supervised training paradigm is able to outperform fully supervised seq2seq models with less than 10% annotations. By utilizing all annotated data, our model can boost the performance of a standard seq2seq model by over 5 BLEU points.
  arXiv  Detail & Related papers  (2021-02-06T10:21:48Z)
• Data-to-Text Generation with Iterative Text Editing [3.42658286826597]
  We present a novel approach to data-to-text generation based on iterative text editing. We first transform data items to text using trivial templates, and then we iteratively improve the resulting text by a neural model trained for the sentence fusion task. The output of the model is filtered by a simple and reranked with an off-the-shelf pre-trained language model.
  arXiv  Detail & Related papers  (2020-11-03T13:32:38Z)
• Text Editing by Command [82.50904226312451]
  A prevailing paradigm in neural text generation is one-shot generation, where text is produced in a single step. We address this limitation with an interactive text generation setting in which the user interacts with the system by issuing commands to edit existing text. We show that our Interactive Editor, a transformer-based model trained on this dataset, outperforms baselines and obtains positive results in both automatic and human evaluations.
  arXiv  Detail & Related papers  (2020-10-24T08:00:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.