CUDRT: Benchmarking the Detection of Human vs. Large Language Models Generated Texts

• URL: http://arxiv.org/abs/2406.09056v1
• Date: Thu, 13 Jun 2024 12:43:40 GMT
• Title: CUDRT: Benchmarking the Detection of Human vs. Large Language Models Generated Texts
• Authors: Zhen Tao, Zhiyu Li, Dinghao Xi, Wei Xu,
• Abstract summary: This paper constructs a comprehensive benchmark in both Chinese and English to evaluate mainstream AI-generated text detectors. We categorize text generation into five distinct operations: Create, Update, Delete, Rewrite, and Translate. For each CUDRT category, we have developed extensive datasets to thoroughly assess detector performance.
• Score: 10.027843402296678
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The proliferation of large language models (LLMs) has significantly enhanced text generation capabilities across various industries. However, these models' ability to generate human-like text poses substantial challenges in discerning between human and AI authorship. Despite the effectiveness of existing AI-generated text detectors, their development is hindered by the lack of comprehensive, publicly available benchmarks. Current benchmarks are limited to specific scenarios, such as question answering and text polishing, and predominantly focus on English texts, failing to capture the diverse applications and linguistic nuances of LLMs. To address these limitations, this paper constructs a comprehensive bilingual benchmark in both Chinese and English to evaluate mainstream AI-generated text detectors. We categorize LLM text generation into five distinct operations: Create, Update, Delete, Rewrite, and Translate (CUDRT), encompassing all current LLMs activities. We also establish a robust benchmark evaluation framework to support scalable and reproducible experiments. For each CUDRT category, we have developed extensive datasets to thoroughly assess detector performance. By employing the latest mainstream LLMs specific to each language, our datasets provide a thorough evaluation environment. Extensive experimental results offer critical insights for optimizing AI-generated text detectors and suggest future research directions to improve detection accuracy and generalizability across various scenarios.

Related papers

• DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios [38.952481877244644]
  We present a new benchmark, DetectRL, highlighting that even state-of-the-art (SOTA) detection techniques still underperformed in this task. Our development of DetectRL reveals the strengths and limitations of current SOTA detectors. We believe DetectRL could serve as an effective benchmark for assessing detectors in real-world scenarios.
  arXiv  Detail & Related papers  (2024-10-31T09:01:25Z)
• GigaCheck: Detecting LLM-generated Content [72.27323884094953]
  In this work, we investigate the task of generated text detection by proposing the GigaCheck. Our research explores two approaches: (i) distinguishing human-written texts from LLM-generated ones, and (ii) detecting LLM-generated intervals in Human-Machine collaborative texts. Specifically, we use a fine-tuned general-purpose LLM in conjunction with a DETR-like detection model, adapted from computer vision, to localize artificially generated intervals within text.
  arXiv  Detail & Related papers  (2024-10-31T08:30:55Z)
• Unveiling Large Language Models Generated Texts: A Multi-Level Fine-Grained Detection Framework [9.976099891796784]
  Large language models (LLMs) have transformed human writing by enhancing grammar correction, content expansion, and stylistic refinement. Existing detection methods, which mainly rely on single-feature analysis and binary classification, often fail to effectively identify LLM-generated text in academic contexts. We propose a novel Multi-level Fine-grained Detection framework that detects LLM-generated text by integrating low-level structural, high-level semantic, and deep-level linguistic features.
  arXiv  Detail & Related papers  (2024-10-18T07:25:00Z)
• Detecting Machine-Generated Long-Form Content with Latent-Space Variables [54.07946647012579]
  Existing zero-shot detectors primarily focus on token-level distributions, which are vulnerable to real-world domain shifts. We propose a more robust method that incorporates abstract elements, such as event transitions, as key deciding factors to detect machine versus human texts.
  arXiv  Detail & Related papers  (2024-10-04T18:42:09Z)
• Spotting AI's Touch: Identifying LLM-Paraphrased Spans in Text [61.22649031769564]
  We propose a novel framework, paraphrased text span detection (PTD) PTD aims to identify paraphrased text spans within a text. We construct a dedicated dataset, PASTED, for paraphrased text span detection.
  arXiv  Detail & Related papers  (2024-05-21T11:22:27Z)
• ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detection [6.27025292177391]
  ToBlend is a novel token-level ensemble text generation method to challenge the robustness of current AI-content detection approaches. We find ToBlend significantly drops the performance of most mainstream AI-content detection methods.
  arXiv  Detail & Related papers  (2024-02-17T02:25:57Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• The Imitation Game: Detecting Human and AI-Generated Texts in the Era of ChatGPT and BARD [3.2228025627337864]
  We introduce a novel dataset of human-written and AI-generated texts in different genres. We employ several machine learning models to classify the texts. Results demonstrate the efficacy of these models in discerning between human and AI-generated text.
  arXiv  Detail & Related papers  (2023-07-22T21:00:14Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.