TM-TREK at SemEval-2024 Task 8: Towards LLM-Based Automatic Boundary Detection for Human-Machine Mixed Text

• URL: http://arxiv.org/abs/2404.00899v1
• Date: Mon, 1 Apr 2024 03:54:42 GMT
• Title: TM-TREK at SemEval-2024 Task 8: Towards LLM-Based Automatic Boundary Detection for Human-Machine Mixed Text
• Authors: Xiaoyan Qu, Xiangfeng Meng,
• Abstract summary: This paper explores the ability of large language models to identify boundaries in human-written and machine-generated mixed texts. Our ensemble model of LLMs achieved first place in the 'Human-Machine Mixed Text Detection' sub-task of the SemEval'24 Competition Task 8.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the increasing prevalence of text generated by large language models (LLMs), there is a growing concern about distinguishing between LLM-generated and human-written texts in order to prevent the misuse of LLMs, such as the dissemination of misleading information and academic dishonesty. Previous research has primarily focused on classifying text as either entirely human-written or LLM-generated, neglecting the detection of mixed texts that contain both types of content. This paper explores LLMs' ability to identify boundaries in human-written and machine-generated mixed texts. We approach this task by transforming it into a token classification problem and regard the label turning point as the boundary. Notably, our ensemble model of LLMs achieved first place in the 'Human-Machine Mixed Text Detection' sub-task of the SemEval'24 Competition Task 8. Additionally, we investigate factors that influence the capability of LLMs in detecting boundaries within mixed texts, including the incorporation of extra layers on top of LLMs, combination of segmentation loss, and the impact of pretraining. Our findings aim to provide valuable insights for future research in this area.

Related papers

• Understanding the Effects of Human-written Paraphrases in LLM-generated Text Detection [7.242609314791262]
  Human & LLM Paraphrase Collection (HLPC) is a first-of-its-kind dataset that incorporates human-written texts and paraphrases. We perform classification experiments that incorporate human-written paraphrases, watermarked and non-watermarked LLM-generated documents from GPT and OPT, and LLM-generated paraphrases from DIPPER and BART. Results show that the inclusion of human-written paraphrases has a significant impact of LLM-generated detector performance, promoting TPR@1%FPR with a possible trade-off of AUROC and accuracy.
  arXiv  Detail & Related papers  (2024-11-06T10:06:21Z)
• GigaCheck: Detecting LLM-generated Content [72.27323884094953]
  In this work, we investigate the task of generated text detection by proposing the GigaCheck. Our research explores two approaches: (i) distinguishing human-written texts from LLM-generated ones, and (ii) detecting LLM-generated intervals in Human-Machine collaborative texts. Specifically, we use a fine-tuned general-purpose LLM in conjunction with a DETR-like detection model, adapted from computer vision, to localize artificially generated intervals within text.
  arXiv  Detail & Related papers  (2024-10-31T08:30:55Z)
• Beyond Binary: Towards Fine-Grained LLM-Generated Text Detection via Role Recognition and Involvement Measurement [51.601916604301685]
  Large language models (LLMs) generate content that can undermine trust in online discourse. Current methods often focus on binary classification, failing to address the complexities of real-world scenarios like human-AI collaboration. To move beyond binary classification and address these challenges, we propose a new paradigm for detecting LLM-generated content.
  arXiv  Detail & Related papers  (2024-10-18T08:14:10Z)
• Peering into the Mind of Language Models: An Approach for Attribution in Contextual Question Answering [9.86691461253151]
  We introduce a novel method for attribution in contextual question answering, leveraging the hidden state representations of large language models (LLMs) Our approach bypasses the need for extensive model retraining and retrieval model overhead, offering granular attributions and preserving the quality of generated answers. We present Verifiability-granular, an attribution dataset which has token level annotations for LLM generations in the contextual question answering setup.
  arXiv  Detail & Related papers  (2024-05-28T09:12:44Z)
• ReMoDetect: Reward Models Recognize Aligned LLM's Generations [55.06804460642062]
  Large language models (LLMs) generate human-preferable texts. In this paper, we identify the common characteristics shared by these models. We propose two training schemes to further improve the detection ability of the reward model.
  arXiv  Detail & Related papers  (2024-05-27T17:38:33Z)
• LLM-Detector: Improving AI-Generated Chinese Text Detection with Open-Source LLM Instruction Tuning [4.328134379418151]
  Existing AI-generated text detection models are prone to in-domain over-fitting. We propose LLM-Detector, a novel method for both document-level and sentence-level text detection.
  arXiv  Detail & Related papers  (2024-02-02T05:54:12Z)
• A Survey on Detection of LLMs-Generated Content [97.87912800179531]
  The ability to detect LLMs-generated content has become of paramount importance. We aim to provide a detailed overview of existing detection strategies and benchmarks. We also posit the necessity for a multi-faceted approach to defend against various attacks.
  arXiv  Detail & Related papers  (2023-10-24T09:10:26Z)
• A Survey on LLM-Generated Text Detection: Necessity, Methods, and Future Directions [39.36381851190369]
  There is an imperative need to develop detectors that can detect LLM-generated text. This is crucial to mitigate potential misuse of LLMs and safeguard realms like artistic expression and social networks from harmful influence of LLM-generated content. The detector techniques have witnessed notable advancements recently, propelled by innovations in watermarking techniques, statistics-based detectors, neural-base detectors, and human-assisted methods.
  arXiv  Detail & Related papers  (2023-10-23T09:01:13Z)
• SeqXGPT: Sentence-Level AI-Generated Text Detection [62.3792779440284]
  We introduce a sentence-level detection challenge by synthesizing documents polished with large language models (LLMs) We then propose textbfSequence textbfX (Check) textbfGPT, a novel method that utilizes log probability lists from white-box LLMs as features for sentence-level AIGT detection.
  arXiv  Detail & Related papers  (2023-10-13T07:18:53Z)
• MAGE: Machine-generated Text Detection in the Wild [82.70561073277801]
  Large language models (LLMs) have achieved human-level text generation, emphasizing the need for effective AI-generated text detection. We build a comprehensive testbed by gathering texts from diverse human writings and texts generated by different LLMs. Despite challenges, the top-performing detector can identify 86.54% out-of-domain texts generated by a new LLM, indicating the feasibility for application scenarios.
  arXiv  Detail & Related papers  (2023-05-22T17:13:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.