When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection

• URL: http://arxiv.org/abs/2608.29903v1
• Date: Sun, 30 Aug 2026 17:00:06 GMT
• Title: When Less is More: Understanding When Token Filtering Helps and Fails in AI-generated Text Detection
• Abstract summary: We introduce top-$k$ cumulative probability filtering as a diagnostic probe.<n>We find that filtering helps for weak source LMs, where low-entropy tokens are harmful, but fails for strong source LMs, where they are not notably harmful.<n>Our work provides the first systematic analysis showing that some tokens are not merely uninformative but systematically harmful due to entropy miscalibration, revealing a two-sided trade-off in token-level detection.
• Score: 11.142641505070207
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid advancement of large language models (LLMs) has made AI-generated text detection increasingly critical. Existing zero-shot detectors assume that more token-level evidence leads to more reliable detection. However, our empirical study challenges this consensus: fewer tokens sometimes work better, retaining only 40% can yield optimal performance, yet this benefit is not universal. Using the Entropy Gap Score (EGS), we introduce top-$k$ cumulative probability filtering as a diagnostic probe. Across three representative settings, filtering exhibits strikingly different behaviors. We analyze EGS via typical set theory and quantify its dynamics through entropy calibration and distribution analysis. We find that filtering helps for weak source LMs, where low-entropy tokens are harmful, but fails for strong source LMs, where they are not notably harmful. Our work provides the first systematic analysis showing that some tokens are not merely uninformative but systematically harmful due to entropy miscalibration, revealing a two-sided trade-off in token-level detection.

Related papers

• On the Salience of Low-Probability Tokens for AI-Generated Text Detection: A Multiscale Uncertainty Perspective [19.794084116453146]
  Uncertainty is a multiscale uncertainty estimator that focuses on informative low-probability tokens.<n>It alleviates boilerplate dominance by averaging the log-probabilities of low-probability tokens.<n>It reduces brittleness by capturing the distributional shape of this low-probability region via Rényi entropy.
  arXiv  Detail & Related papers  (2026-06-01T12:21:41Z)
• Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens [9.36751822006717]
  We propose Entropy-guided Token Weighting (ETW) as a token-level unlearning regularizer.<n>ETW uses entropy of the predictive distribution as a proxy for token informativeness.<n>We demonstrate that informative tokens tend to have higher entropy, whereas structural tokens tend to have lower entropy.
  arXiv  Detail & Related papers  (2026-04-20T04:20:29Z)
• Segment-Level Coherence for Robust Harmful Intent Probing in LLMs [13.0633878066404]
  We introduce a streaming probing objective that requires multiple evidence tokens to consistently support a prediction.<n>At a fixed 1% false-positive rate, our method improves the true-positive rate by 35.55%.<n>Even when adversarial fine-tuning enables novel character-level ciphers, harmful intent remains detectable.
  arXiv  Detail & Related papers  (2026-04-16T10:56:40Z)
• Exons-Detect: Identifying and Amplifying Exonic Tokens via Hidden-State Discrepancy for Robust AI-Generated Text Detection [18.509745023735835]
  We propose Exons-Detect, a training-free method for AI-generated text detection based on an exon-aware token reweighting perspective.<n>Exons-Detect identifies and amplifies informative exonic tokens by measuring hidden-state discrepancy under a dual-model setting, and computes an interpretable translation score from the resulting importance-weighted token sequence.
  arXiv  Detail & Related papers  (2026-03-26T03:12:19Z)
• Diversity Boosts AI-Generated Text Detection [51.56484100374058]
  DivEye is a novel framework that captures how unpredictability fluctuates across a text using surprisal-based features.<n>Our method outperforms existing zero-shot detectors by up to 33.2% and achieves competitive performance with fine-tuned baselines.
  arXiv  Detail & Related papers  (2025-09-23T10:21:22Z)
• Character-Level Perturbations Disrupt LLM Watermarks [64.60090923837701]
  We formalize the system model for Large Language Model (LLM) watermarking.<n>We characterize two realistic threat models constrained on limited access to the watermark detector.<n>We demonstrate character-level perturbations are significantly more effective for watermark removal under the most restrictive threat model.<n> Experiments confirm the superiority of character-level perturbations and the effectiveness of the Genetic Algorithm (GA) in removing watermarks under realistic constraints.
  arXiv  Detail & Related papers  (2025-09-11T02:50:07Z)
• Invisible Entropy: Towards Safe and Efficient Low-Entropy LLM Watermarking [48.26359966929394]
  Invisible Entropy (IE) is a watermarking paradigm designed to enhance both safety and efficiency.<n>IE reduces parameter size by 99% while achieving performance on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-05-20T09:19:06Z)
• Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection [58.419940585826744]
  We introduce FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors.<n>We partitioned data into subgroups based on attributes (e.g., text length and writing style) and implemented FairOPT to learn decision thresholds for each group to reduce discrepancy.<n>Our framework paves the way for more robust classification in AI-generated content detection via post-processing.
  arXiv  Detail & Related papers  (2025-02-06T21:58:48Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Toxicity Detection for Free [16.07605369484645]
  We introduce Moderation Using LLM Introspection (MULI), which detects toxic prompts using the information extracted directly from LLMs themselves. We build a robust detector of toxic prompts using a sparse logistic regression model on the first response token logits.
  arXiv  Detail & Related papers  (2024-05-29T07:03:31Z)
• Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information [67.78183175605761]
  Large Language Models are susceptible to adversarial prompt attacks. This vulnerability underscores a significant concern regarding the robustness and reliability of LLMs. We introduce a novel approach to detecting adversarial prompts at a token level.
  arXiv  Detail & Related papers  (2023-11-20T03:17:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.