How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study

• URL: http://arxiv.org/abs/2510.13681v1
• Date: Wed, 15 Oct 2025 15:36:45 GMT
• Title: How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study
• Authors: Matthieu Dubois, François Yvon, Pablo Piantanida,
• Abstract summary: Large Language Models (LLMs) are increasingly common and often indistinguishable from human-written content.<n>Many recent detectors report near-perfect accuracy, often boasting AUROC scores above 99%.<n>In this work, we examine how sampling-based decoding impacts detectability.
• Score: 39.866323800060066
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As texts generated by Large Language Models (LLMs) are ever more common and often indistinguishable from human-written content, research on automatic text detection has attracted growing attention. Many recent detectors report near-perfect accuracy, often boasting AUROC scores above 99\%. However, these claims typically assume fixed generation settings, leaving open the question of how robust such systems are to changes in decoding strategies. In this work, we systematically examine how sampling-based decoding impacts detectability, with a focus on how subtle variations in a model's (sub)word-level distribution affect detection performance. We find that even minor adjustments to decoding parameters - such as temperature, top-p, or nucleus sampling - can severely impair detector accuracy, with AUROC dropping from near-perfect levels to 1\% in some settings. Our findings expose critical blind spots in current detection methods and emphasize the need for more comprehensive evaluation protocols. To facilitate future research, we release a large-scale dataset encompassing 37 decoding configurations, along with our code and evaluation framework https://github.com/BaggerOfWords/Sampling-and-Detection

Related papers

• Human Texts Are Outliers: Detecting LLM-generated Texts via Out-of-distribution Detection [71.59834293521074]
  We develop a framework to distinguish between human-authored and machine-generated text.<n>Our method achieves 98.3% AUROC and AUPR with only 8.9% FPR95 on DeepFake dataset.<n>Code, pretrained weights, and demo will be released.
  arXiv  Detail & Related papers  (2025-10-07T08:14:45Z)
• Text-ADBench: Text Anomaly Detection Benchmark based on LLMs Embedding [27.02879006439693]
  This work performs a comprehensive empirical study and introduces a benchmark for text anomaly detection.<n>Our work systematically evaluates the effectiveness of embedding-based text anomaly detection.<n>By open-sourcing our benchmark toolkit, this work provides a foundation for future research in robust and scalable text anomaly detection systems.
  arXiv  Detail & Related papers  (2025-07-16T14:47:41Z)
• TempTest: Local Normalization Distortion and the Detection of Machine-generated Text [0.0]
  We introduce a method for detecting machine-generated text that is entirely of the generating language model.<n>This is achieved by targeting a defect in the way that decoding strategies, such as temperature or top-k sampling, normalize conditional probability measures.<n>We evaluate our detector in the white and black box settings across various language models, datasets, and passage lengths.
  arXiv  Detail & Related papers  (2025-03-26T10:56:59Z)
• Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection [58.419940585826744]
  We introduce FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors.<n>We partitioned data into subgroups based on attributes (e.g., text length and writing style) and implemented FairOPT to learn decision thresholds for each group to reduce discrepancy.<n>Our framework paves the way for more robust classification in AI-generated content detection via post-processing.
  arXiv  Detail & Related papers  (2025-02-06T21:58:48Z)
• A Practical Examination of AI-Generated Text Detectors for Large Language Models [25.919278893876193]
  Machine-generated content detectors claim to identify such text under various conditions and from any language model.<n>This paper critically evaluates these claims by assessing several popular detectors on a range of domains, datasets, and models that these detectors have not previously encountered.
  arXiv  Detail & Related papers  (2024-12-06T15:56:11Z)
• DetectRL: Benchmarking LLM-Generated Text Detection in Real-World Scenarios [38.952481877244644]
  We present a new benchmark, DetectRL, highlighting that even state-of-the-art (SOTA) detection techniques still underperformed in this task.<n>Using popular large language models (LLMs), we generated data that better aligns with real-world applications.<n>We analyzed the potential impact of writing styles, model types, attack methods, the text lengths, and real-world human writing factors on different types of detectors.
  arXiv  Detail & Related papers  (2024-10-31T09:01:25Z)
• Assaying on the Robustness of Zero-Shot Machine-Generated Text Detectors [57.7003399760813]
  We explore advanced Large Language Models (LLMs) and their specialized variants, contributing to this field in several ways. We uncover a significant correlation between topics and detection performance. These investigations shed light on the adaptability and robustness of these detection methods across diverse topics.
  arXiv  Detail & Related papers  (2023-12-20T10:53:53Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• Can AI-Generated Text be Reliably Detected? [50.95804851595018]
  Large Language Models (LLMs) perform impressively well in various applications.<n>The potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use.<n>We stress-test the robustness of these AI text detectors in the presence of an attacker.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.