Log-Likelihood, Simpson's Paradox, and the Detection of Machine-Generated Text

• URL: http://arxiv.org/abs/2605.06294v1
• Date: Thu, 07 May 2026 13:59:07 GMT
• Title: Log-Likelihood, Simpson's Paradox, and the Detection of Machine-Generated Text
• Abstract summary: We show that the token-level signal distinguishing human and machine text is non-uniform across the hidden space of the detector model.<n>We introduce a learned local calibration step grounded in Bayesian decision theory.<n>This single intervention dramatically and consistently improves detection performance across all baseline detectors and all datasets we consider.
• Score: 0.039998518782208774
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The ability to reliably distinguish human-written text from that generated by large language models is of profound societal importance. The dominant approach to this problem exploits the likelihood hypothesis: that machine-generated text should appear more probable to a detector language model than human-written text. However, we demonstrate that the token-level signal distinguishing human and machine text is non-uniform across the hidden space of the detector model, and naively averaging likelihood-based token scores across regions with fundamentally different statistical structure, as most detectors do, causes a form of Simpson's paradox: a strong local signal is destroyed by inappropriate aggregation. To correct for this, we introduce a learned local calibration step grounded in Bayesian decision theory. Rather than aggregating raw token scores, we first learn lightweight predictors of the score distributions conditioned on position in hidden space, and aggregate calibrated log-likelihood ratios instead. This single intervention dramatically and consistently improves detection performance across all baseline detectors and all datasets we consider. For example, our calibrated variant of Fast-DetectGPT improves AUROC from $0.63$ to $0.85$ on GPT-5.4 text, and a locally-calibrated DMAP detector we introduce achieves state-of-the-art performance across the board. That said, our central contribution is not a new detector, but a precise diagnosis of a significant cause of under-performance of existing detectors and a principled, modular remedy compatible with any token-averaging pipeline. This will serve as a foundation for the community to build upon, with natural avenues including richer distributional models, improved calibration strategies, and principled ensembling with hidden-space geometry signals via the full Bayes-optimal decision rule.

Related papers

• Rethinking AI-Generated Text Detection: A Strong Baseline and the Distribution-Shift Problem That Remains [27.078441680572254]
  We show that a plain, fully fine-tuned RoBERTa matches or exceeds the specialized detectors those benchmarks are built around.<n>The remaining challenge is the distribution shift.<n>We identify a key failure mode: under distribution shift, the detector can assign high-confidence machine labels to human-written text.
  arXiv  Detail & Related papers  (2026-07-04T03:20:23Z)
• Base Models Look Human To AI Detectors [68.7723287560362]
  We show that generated text from base models is often judged overwhelmingly human, whereas text generated by their instruction-tuned counterparts is not.<n>We propose Humanization by Iterative Paraphrasing (HIP), a detector-agnostic pipeline that minimally fine-tunes a base model into a paraphraser and applies it iteratively.<n>Our findings suggest that current detectors are tracking artifacts of instruction tuning and local context more than any in notion of machine-generated text.
  arXiv  Detail & Related papers  (2026-05-19T08:13:12Z)
• Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy [51.887915969023965]
  We show that modern Large Language Models (LLMs) undergo alignment, leaving a measurable distributional imprint.<n>To mitigate the instability in high-entropy regions, we introduce Log-likelihood Alignment Preference Discrepancy (LAPD)<n>LAPD is a standardized information-weighted statistic based on alignment imprint.
  arXiv  Detail & Related papers  (2026-04-18T09:12:24Z)
• Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection [105.14032334647932]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, highlighting the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods that are prone to overfitting.<n>We propose a Markov-informed score calibration strategy that models two relationships of context detection scores that may aid calibration.
  arXiv  Detail & Related papers  (2026-02-08T16:06:12Z)
• How Sampling Affects the Detectability of Machine-written texts: A Comprehensive Study [39.866323800060066]
  Large Language Models (LLMs) are increasingly common and often indistinguishable from human-written content.<n>Many recent detectors report near-perfect accuracy, often boasting AUROC scores above 99%.<n>In this work, we examine how sampling-based decoding impacts detectability.
  arXiv  Detail & Related papers  (2025-10-15T15:36:45Z)
• When Personalization Tricks Detectors: The Feature-Inversion Trap in Machine-Generated Text Detection [64.23509202768945]
  We introduce dataset, the first benchmark for evaluating detector robustness in personalized settings.<n>Our experimental results demonstrate large performance gaps across detectors in personalized settings.<n>We propose method, a simple and reliable way to predict detector performance changes in personalized settings.
  arXiv  Detail & Related papers  (2025-10-14T13:10:23Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• T-Detect: Tail-Aware Statistical Normalization for Robust Detection of Adversarial Machine-Generated Text [16.983417848762826]
  Large language models (LLMs) have shown the capability to generate fluent and logical content, presenting significant challenges to machine-generated text detection.<n>In this paper, we introduce T-Detect, a novel detection method that fundamentally redesigns the curvature-based detectors.<n>Our contributions are a new, theoretically-justified statistical foundation for text detection, an ablation-validated method that demonstrates superior robustness, and a comprehensive analysis of its performance under adversarial conditions.
  arXiv  Detail & Related papers  (2025-07-31T14:08:04Z)
• Language Models Optimized to Fool Detectors Still Have a Distinct Style (And How to Change It) [5.789169343514737]
  stylistic feature space may be used to reliably detect samples from language models optimized to prevent detection.<n>We explore a new paraphrasing approach that simultaneously aims to close the gap between human writing and machine writing in stylistic feature space.
  arXiv  Detail & Related papers  (2025-05-20T16:55:44Z)
• TempTest: Local Normalization Distortion and the Detection of Machine-generated Text [0.0]
  We introduce a method for detecting machine-generated text that is entirely of the generating language model.<n>This is achieved by targeting a defect in the way that decoding strategies, such as temperature or top-k sampling, normalize conditional probability measures.<n>We evaluate our detector in the white and black box settings across various language models, datasets, and passage lengths.
  arXiv  Detail & Related papers  (2025-03-26T10:56:59Z)
• Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature [33.84718903572477]
  Large language models (LLMs) have shown the ability to produce fluent and cogent content.<n>To build trustworthy AI systems, it is imperative to distinguish between machine-generated and human-authored content.<n>Fast-DetectGPT is an optimized zero-shot detector that substitutes DetectGPT's perturbation step with a more efficient sampling step.
  arXiv  Detail & Related papers  (2023-10-08T11:41:28Z)
• DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature [143.5381108333212]
  We show that text sampled from an large language model tends to occupy negative curvature regions of the model's log probability function. We then define a new curvature-based criterion for judging if a passage is generated from a given LLM. We find DetectGPT is more discriminative than existing zero-shot methods for model sample detection.
  arXiv  Detail & Related papers  (2023-01-26T18:44:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.