TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices

• URL: http://arxiv.org/abs/2605.31113v1
• Date: Fri, 29 May 2026 10:25:32 GMT
• Title: TSM-Bench: Detecting LLM-Generated Text in Real-World Wikipedia Editing Practices
• Abstract summary: Existing detection benchmarks primarily focus on textitgeneric text generation tasks.<n> editors frequently employ LLMs for specific writing tasks.<n>These textittask-specific MGT instances tend to resemble human-written text more closely due to their constrained task formulation and contextual conditioning.<n>We introduce textscTSM-Bench, a multilingual, multi-generator, and textitmulti-task benchmark for evaluating MGT detectors on common, real-world Wikipedia editing tasks.
• Score: 3.304863806104706
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatically detecting machine-generated text (MGT) is critical to maintaining the knowledge integrity of user-generated content (UGC) platforms such as Wikipedia. Existing detection benchmarks primarily focus on \textit{generic} text generation tasks (e.g., ``Write an article about machine learning.''). However, editors frequently employ LLMs for specific writing tasks (e.g., summarisation). These \textit{task-specific} MGT instances tend to resemble human-written text more closely due to their constrained task formulation and contextual conditioning. In this work, we show that a range of SOTA MGT detectors struggle to identify task-specific MGT reflecting real-world editing on Wikipedia. We introduce \textsc{TSM-Bench}, a multilingual, multi-generator, and \textit{multi-task} benchmark for evaluating MGT detectors on common, real-world Wikipedia editing tasks. Our findings demonstrate that (\textit{i}) average detection accuracy drops by 10--40\% compared to prior benchmarks, and (\textit{ii}) a generalisation asymmetry exists: fine-tuning on task-specific data enables generalisation to generic data -- even across domains -- but not vice versa. We demonstrate that models fine-tuned exclusively on generic MGT overfit to superficial artefacts of machine generation. Our results suggest that, in contrast to prior benchmarks, most detectors remain unreliable for automated detection in real-world contexts such as UGC platforms. \textsc{TSM-Bench} therefore provides a critical foundation for developing and evaluating future models.

Related papers

• Hidden Human-Like Nature of Machine-Generated Texts: Theory and Detection Enhancement [108.30620357325559]
  Machine-generated texts (MGTs) produced by large language models (LLMs) are increasingly prevalent across various applications.<n>Existing paragraph-level detection methods treat MGTs as entirely machine-like, overlooking the hidden human-like nature of MGTs.<n>We propose a model-agnostic stacked enhancement framework that improves existing detectors by reducing the influence of hidden human-like spans.
  arXiv  Detail & Related papers  (2026-05-22T03:17:21Z)
• GISA: A Benchmark for General Information-Seeking Assistant [102.30831921333755]
  GISA is a benchmark for General Information-Seeking Assistants comprising 373 human-crafted queries.<n>It integrates both deep reasoning and broad information aggregation within unified tasks, and includes a live subset with periodically updated answers to resist memorization.<n>Experiments on mainstream LLMs and commercial search products reveal that even the best-performing model achieves only 19.30% exact match score.
  arXiv  Detail & Related papers  (2026-02-09T11:44:15Z)
• DetectAnyLLM: Towards Generalizable and Robust Detection of Machine-Generated Text Across Domains and Models [60.713908578319256]
  We propose Direct Discrepancy Learning (DDL) to optimize the detector with task-oriented knowledge.<n>Built upon this, we introduce DetectAnyLLM, a unified detection framework that achieves state-of-the-art MGTD performance.<n>MIRAGE samples human-written texts from 10 corpora across 5 text-domains, which are then re-generated or revised using 17 cutting-edge LLMs.
  arXiv  Detail & Related papers  (2025-09-15T10:59:57Z)
• Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling [83.78874399606379]
  We propose MACT, a Multi-Agent Collaboration framework with Test-Time scaling.<n>It comprises four distinct small-scale agents, with clearly defined roles and effective collaboration.<n>It shows superior performance with a smaller parameter scale without sacrificing the ability of general and mathematical tasks.
  arXiv  Detail & Related papers  (2025-08-05T12:52:09Z)
• WETBench: A Benchmark for Detecting Task-Specific Machine-Generated Text on Wikipedia [2.255682336735152]
  Existing work primarily evaluates MGT detectors on generic generation tasks.<n>We introduce a multilingual, multi-generator, and task-specific benchmark for MGT detection.<n>We find that, across settings, training-based detectors achieve an average accuracy of 78%, while zero-shot detectors average 58%.
  arXiv  Detail & Related papers  (2025-07-04T08:13:10Z)
• Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors [4.7713095161046555]
  We present a pipeline to test the resilience of state-of-the-art MGT detectors to linguistically informed adversarial attacks.<n>We fine-tune language models to shift the MGT style toward human-written text (HWT)<n>This exploits the detectors' reliance on stylistic clues, making new generations more challenging to detect.
  arXiv  Detail & Related papers  (2025-05-30T12:33:30Z)
• On the Generalization and Adaptation Ability of Machine-Generated Text Detectors in Academic Writing [23.434925348283617]
  This work investigates the generalization and adaptation capabilities of MGT detectors in three key aspects specific to academic writing.<n>We benchmark the performance of various detectors for binary classification and attribution tasks in both in-domain and cross-domain settings.<n>Our findings provide insights into the generalization and adaptation ability of MGT detectors across diverse scenarios and lay the foundation for building robust, adaptive detection systems.
  arXiv  Detail & Related papers  (2024-12-23T03:30:34Z)
• LLM-DetectAIve: a Tool for Fine-Grained Machine-Generated Text Detection [87.43727192273772]
  It is often hard to tell whether a piece of text was human-written or machine-generated.<n>We present LLM-DetectAIve, designed for fine-grained detection.<n>It supports four categories: (i) human-written, (ii) machine-generated, (iii) machine-written, then machine-humanized, and (iv) human-written, then machine-polished.
  arXiv  Detail & Related papers  (2024-08-08T07:43:17Z)
• M4GT-Bench: Evaluation Benchmark for Black-Box Machine-Generated Text Detection [69.41274756177336]
  Large Language Models (LLMs) have brought an unprecedented surge in machine-generated text (MGT) across diverse channels. This raises legitimate concerns about its potential misuse and societal implications. We introduce a new benchmark based on a multilingual, multi-domain, and multi-generator corpus of MGTs -- M4GT-Bench.
  arXiv  Detail & Related papers  (2024-02-17T02:50:33Z)
• TextMachina: Seamless Generation of Machine-Generated Text Datasets [2.4578723416255754]
  TextMachina is a Python framework designed to aid in the creation of high-quality, unbiased datasets. It provides a user-friendly pipeline that abstracts away the inherent intricacies of building MGT datasets. The quality of the datasets generated by TextMachina has been assessed in previous works.
  arXiv  Detail & Related papers  (2024-01-08T15:05:32Z)
• M4: Multi-generator, Multi-domain, and Multi-lingual Black-Box Machine-Generated Text Detection [69.29017069438228]
  Large language models (LLMs) have demonstrated remarkable capability to generate fluent responses to a wide variety of user queries. This has also raised concerns about the potential misuse of such texts in journalism, education, and academia. In this study, we strive to create automated systems that can detect machine-generated texts and pinpoint potential misuse.
  arXiv  Detail & Related papers  (2023-05-24T08:55:11Z)
• FETA: Towards Specializing Foundation Models for Expert Task Applications [49.57393504125937]
  Foundation Models (FMs) have demonstrated unprecedented capabilities including zero-shot learning, high fidelity data synthesis, and out of domain generalization. We show in this paper that FMs still have poor out-of-the-box performance on expert tasks. We propose a first of its kind FETA benchmark built around the task of teaching FMs to understand technical documentation.
  arXiv  Detail & Related papers  (2022-09-08T08:47:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.