Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race

• URL: http://arxiv.org/abs/2605.02620v1
• Date: Mon, 04 May 2026 14:10:41 GMT
• Title: Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race
• Abstract summary: Reproducing an empirical NLP study used to take weeks.<n>All code, $648$ mimic drafts, trained detectors, diagnostics, and adversarial trajectories are released.
• Score: 3.9508043303559828
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reproducing an empirical NLP study used to take weeks. Given the released data and a modern agentic-research harness, we redo every experiment of a recent ACL\,2026 study on personal-style post-editing of LLM drafts -- and add three new ones -- with the human investigator acting only as a reviewer-in-the-loop. We reproduce all seven preregistered hypotheses and recover the paper's headline correlation between perceived self-similarity and embedding-measured self-similarity to three decimal places ($r{=}{+}0.244$, $p{<}10^{-8}$, $n{=}648$). Under a leakage-free held-out protocol, GPT-5.5 and Claude\,Opus\,4.7 close $71$--$75\,\%$ of the style gap to the same-author ceiling on $324$ paired tasks, against $24\,\%$ for the human post-edit, and beat the human post-edit on $\sim$$80\,\%$ of tasks. We then frame the same data as an AI-text detection arms race. A leave-authors-out linear SVM on LUAR-MUD embeddings reaches AUC $0.93$--$1.00$ across approaches; six diagnostics show that GPT-5.5 detection is mostly a length confound while Opus detection is a genuine stylistic signature. Given $T{=}20$ feedback iterations against the frozen detector, an Opus agent flips two of five held-out test mimics to the human half-space and shrinks every margin by an order of magnitude. With moderate effort against a known detector, a frontier LLM can already efficiently lower its own AI-detection probability. All code, $648$ mimic drafts, trained detectors, diagnostics, and adversarial trajectories are released.

Related papers

• Inter-Rater Reliability of LLM and Rule-Based Annotation for Inferential Narrative Features: Three Studies on a Turkish Corpus [0.0]
  This report answers the question of whether a human would agree with labels in a dataset with automatically generated feature annotations.<n>Three studies are reported. Five machine labellers returned positive rates of $0$, $1$, $40$, $72$ and $78$ out of $100$ scenes, against a human count of $9$.<n>Two readings survive: the feature is genuinely inferential and beyond current automatic detection, or the rule's definition is not yet operational enough for any rater to apply consistently.
  arXiv  Detail & Related papers  (2026-09-12T13:31:30Z)
• Workload Identification with Physical Side Channels for AI Governance [0.5013248430919224]
  We show that an external observer can identify the class of the workload running on an NVIDIA H200 from its power draw.<n>Four evasion strategies are tested to disguise training as inference, producing an additional 680 adversarial traces.
  arXiv  Detail & Related papers  (2026-08-31T19:59:35Z)
• Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs [52.149036302760386]
  Large language models can be fine-tuned to encode prompt-borne secrets into fluent, seemingly benign outputs.<n>Recent work proposes detection using linear probes that recover the secret from internal activations.<n>We show that this defense can be systematically evaded, but that detectability can be recovered through a targeted data-level intervention.
  arXiv  Detail & Related papers  (2026-06-08T12:27:11Z)
• Hallucination Is Linearly Decodable from Mid-Layer Hidden States in Quantized LLMs [5.584060970507507]
  We investigate whether open-source LLMs encode a linearly separable signal in their hidden states, and at which network depth this signal is strongest.<n>We extract per-layer hidden states on four benchmarks (TruthfulQA, HaluEval-QA, FEVER, and a controlled synthetic set)<n>Peak probing layers fall in a consistent band across model families on natural-language benchmarks.
  arXiv  Detail & Related papers  (2026-05-30T02:04:30Z)
• The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next [0.0]
  Leaderboards rank frontier models on independent axes but do not reveal whether capabilities reinforce or trade off across releases.<n>We decompose paired SWE-bench and GPQA Diamond scores into a population coupling trend and per-release residual.<n>We provide a three-level playbook (locate, diagnose, rotate), a per-lab measurement-priority table, and seven falsifiable predictions.
  arXiv  Detail & Related papers  (2026-05-13T03:19:38Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• MEMSAD: Gradient-Coupled Anomaly Detection for Memory Poisoning in Retrieval-Augmented Agents [0.0]
  We formalize memory poisoning attacks on retrieval-augmented agents as a Stackelberg game with a unified evaluation framework.<n>We show faithful evaluation increases measured attack success by $4times$ (ASR-R: 0.25 to 1.00$)<n>Our primary contribution is MEMSAD, a calibration-based defense grounded in a gradient coupling.
  arXiv  Detail & Related papers  (2026-05-05T08:15:41Z)
• Do We Still Need Humans in the Loop? Comparing Human and LLM Annotation in Active Learning for Hostility Detection [68.37351671559675]
  Active learning can annotate thousands of instances from a short prompt at negligible cost.<n>This raises two questions for active learning (AL): can LLM labels replace human labels within the AL loop, and does AL remain necessary when entire corpora can be labelled at once?<n>We investigate both questions on a new dataset of 277,902 German political TikTok comments.
  arXiv  Detail & Related papers  (2026-04-15T14:10:58Z)
• Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements [66.94250413799232]
  We study mean estimation of a random vector $X$ in a distributed parameter-server-worker setup.<n>Key challenges are adversarial measurements and asynchrony.<n>Results provide a unified finite-time characterization of robustness, identifiability, and statistical efficiency in distributed linear estimation with adversarial workers.
  arXiv  Detail & Related papers  (2026-04-07T11:45:55Z)
• Criterion-referenceability determines LLM-as-a-judge validity across physics assessment formats [0.01116979912801043]
  We compare GPT-5.2, Grok 4.1, Claude Opus 4.5, DeepSeek-V3.2, Gemini Pro 3, and committee aggregations against human markers under blind, solution-provided, false-solution, and exemplar-anchored conditions.<n>For $n=771$ blind university exam questions, models achieve fractional mean absolute errors (fMAE) $approx 0.22$ with robust discriminative validity.<n>Across $n=55$ scripts, blind AI marking is harsher and more variable than human marking, with discriminative validity already poor.
  arXiv  Detail & Related papers  (2026-03-16T02:09:06Z)
• Spectral Sentinel: Scalable Byzantine-Robust Decentralized Federated Learning via Sketched Random Matrix Theory on Blockchain [0.0]
  Byzantine clients poison gradients under heterogeneous (Non-IID) data.<n>We propose Spectral Sentinel, a Byzantine detection and aggregation framework.<n>We implement the full system with blockchain integration on Polygon networks.
  arXiv  Detail & Related papers  (2025-12-14T09:43:03Z)
• $\texttt{Droid}$: A Resource Suite for AI-Generated Code Detection [75.6327970381944]
  $textbf$textttDroidCollection$$ is an open data suite for training and evaluating machine-generated code detectors.<n>It includes over a million code samples, seven programming languages, outputs from 43 coding models, and three real-world coding domains.<n>We also develop a suite of encoder-only detectors trained using a multi-task objective over $textttDroidCollection$$.
  arXiv  Detail & Related papers  (2025-07-11T12:19:06Z)
• Intention Analysis Makes LLMs A Good Jailbreak Defender [79.4014719271075]
  We present a simple yet highly effective defense strategy, i.e., Intention Analysis ($mathbbIA$)<n>$mathbbIA$ works by triggering LLMs' inherent self-correct and improve ability through a two-stage process.<n>Experiments on varying jailbreak benchmarks show that $mathbbIA$ could consistently and significantly reduce the harmfulness in responses.
  arXiv  Detail & Related papers  (2024-01-12T13:15:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.