Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection

• URL: http://arxiv.org/abs/2502.04528v4
• Date: Sat, 24 May 2025 04:44:26 GMT
• Title: Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection
• Authors: Minseok Jung, Cynthia Fuertes Panizo, Liam Dugan, Yi R., Fung, Pin-Yu Chen, Paul Pu Liang,
• Abstract summary: We introduce FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors.<n>Our framework paves the way for more robust classification in AI-generated content detection via post-processing.
• Score: 60.09665704993751
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The advancement of large language models (LLMs) has made it difficult to differentiate human-written text from AI-generated text. Several AI-text detectors have been developed in response, which typically utilize a fixed global threshold (e.g., $\theta = 0.5$) to classify machine-generated text. However, one universal threshold could fail to account for distributional variations by subgroups. For example, when using a fixed threshold, detectors make more false positive errors on shorter human-written text, and more positive classifications of neurotic writing styles among long texts. These discrepancies can lead to misclassifications that disproportionately affect certain groups. We address this critical limitation by introducing FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors. We partitioned data into subgroups based on attributes (e.g., text length and writing style) and implemented FairOPT to learn decision thresholds for each group to reduce discrepancy. In experiments with nine AI text classifiers on three datasets, FairOPT decreases overall balanced error rate (BER) discrepancy by 12\% while minimally sacrificing accuracy by 0.003\%. Our framework paves the way for more robust classification in AI-generated content detection via post-processing.

Related papers

• AuthorMist: Evading AI Text Detectors with Reinforcement Learning [4.806579822134391]
  AuthorMist is a novel reinforcement learning-based system to transform AI-generated text into human-like writing. We show that AuthorMist effectively reduces the detectability of AI-generated text while preserving the original meaning.
  arXiv  Detail & Related papers  (2025-03-10T12:41:05Z)
• Who Writes What: Unveiling the Impact of Author Roles on AI-generated Text Detection [44.05134959039957]
  We investigate how sociolinguistic attributes-gender, CEFR proficiency, academic field, and language environment-impact state-of-the-art AI text detectors. Our results reveal significant biases: CEFR proficiency and language environment consistently affected detector accuracy, while gender and academic field showed detector-dependent effects. These findings highlight the crucial need for socially aware AI text detection to avoid unfairly penalizing specific demographic groups.
  arXiv  Detail & Related papers  (2025-02-18T07:49:31Z)
• ExaGPT: Example-Based Machine-Generated Text Detection for Human Interpretability [62.285407189502216]
  Detecting texts generated by Large Language Models (LLMs) could cause grave mistakes due to incorrect decisions. We introduce ExaGPT, an interpretable detection approach grounded in the human decision-making process. We show that ExaGPT massively outperforms prior powerful detectors by up to +40.9 points of accuracy at a false positive rate of 1%.
  arXiv  Detail & Related papers  (2025-02-17T01:15:07Z)
• Learning to Rewrite: Generalized LLM-Generated Text Detection [19.9477991969521]
  Large language models (LLMs) present significant risks when used to generate non-factual content and spread disinformation at scale. We introduce Learning2Rewrite, a novel framework for detecting AI-generated text with exceptional generalization to unseen domains.
  arXiv  Detail & Related papers  (2024-08-08T05:53:39Z)
• Are AI-Generated Text Detectors Robust to Adversarial Perturbations? [9.001160538237372]
  Current detectors for AI-generated text (AIGT) lack robustness against adversarial perturbations. This paper investigates the robustness of existing AIGT detection methods and introduces a novel detector, the Siamese Calibrated Reconstruction Network (SCRN) The SCRN employs a reconstruction network to add and remove noise from text, extracting a semantic representation that is robust to local perturbations.
  arXiv  Detail & Related papers  (2024-06-03T10:21:48Z)
• Spotting AI's Touch: Identifying LLM-Paraphrased Spans in Text [61.22649031769564]
  We propose a novel framework, paraphrased text span detection (PTD) PTD aims to identify paraphrased text spans within a text. We construct a dedicated dataset, PASTED, for paraphrased text span detection.
  arXiv  Detail & Related papers  (2024-05-21T11:22:27Z)
• Technical Report on the Pangram AI-Generated Text Classifier [0.14732811715354457]
  We present Pangram Text, a transformer-based neural network trained to distinguish text written by large language models from text written by humans. We show that Pangram Text is not biased against nonnative English speakers and generalizes to domains and models unseen during training.
  arXiv  Detail & Related papers  (2024-02-21T17:13:41Z)
• ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detection [6.27025292177391]
  ToBlend is a novel token-level ensemble text generation method to challenge the robustness of current AI-content detection approaches. We find ToBlend significantly drops the performance of most mainstream AI-content detection methods.
  arXiv  Detail & Related papers  (2024-02-17T02:25:57Z)
• SeqXGPT: Sentence-Level AI-Generated Text Detection [62.3792779440284]
  We introduce a sentence-level detection challenge by synthesizing documents polished with large language models (LLMs) We then propose textbfSequence textbfX (Check) textbfGPT, a novel method that utilizes log probability lists from white-box LLMs as features for sentence-level AIGT detection.
  arXiv  Detail & Related papers  (2023-10-13T07:18:53Z)
• Beyond Black Box AI-Generated Plagiarism Detection: From Sentence to Document Level [4.250876580245865]
  Existing AI-generated text classifiers have limited accuracy and often produce false positives. We propose a novel approach using natural language processing (NLP) techniques. We generate multiple paraphrased versions of a given question and inputting them into the large language model to generate answers. By using a contrastive loss function based on cosine similarity, we match generated sentences with those from the student's response.
  arXiv  Detail & Related papers  (2023-06-13T20:34:55Z)
• Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM [31.25193238045053]
  We introduce a novel method, namely GenCo, which leverages the strong generative power of large language models to assist in training a smaller language model. In our method, an LLM plays an important role in the self-training loop of a smaller model in two important ways. It helps crafting additional high-quality training pairs, by rewriting input texts conditioned on predicted labels.
  arXiv  Detail & Related papers  (2023-04-24T07:35:38Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense [56.077252790310176]
  We present a paraphrase generation model (DIPPER) that can paraphrase paragraphs, condition on surrounding context, and control lexical diversity and content reordering. Using DIPPER to paraphrase text generated by three large language models (including GPT3.5-davinci-003) successfully evades several detectors, including watermarking. We introduce a simple defense that relies on retrieving semantically-similar generations and must be maintained by a language model API provider.
  arXiv  Detail & Related papers  (2023-03-23T16:29:27Z)
• Can AI-Generated Text be Reliably Detected? [50.95804851595018]
  Large Language Models (LLMs) perform impressively well in various applications.<n>The potential for misuse of these models in activities such as plagiarism, generating fake news, and spamming has raised concern about their responsible use.<n>We stress-test the robustness of these AI text detectors in the presence of an attacker.
  arXiv  Detail & Related papers  (2023-03-17T17:53:19Z)
• Classifiers are Better Experts for Controllable Text Generation [63.17266060165098]
  We show that the proposed method significantly outperforms recent PPLM, GeDi, and DExperts on PPL and sentiment accuracy based on the external classifier of generated texts. The same time, it is also easier to implement and tune, and has significantly fewer restrictions and requirements.
  arXiv  Detail & Related papers  (2022-05-15T12:58:35Z)
• Efficient text generation of user-defined topic using generative adversarial networks [0.32228025627337864]
  We propose a User-Defined GAN (UD-GAN) with two-level discriminators to solve this problem. The proposed method is capable of generating texts with less time than others.
  arXiv  Detail & Related papers  (2020-06-22T04:49:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.