PersonalBench: Measuring the Authorship Gap in LLM Personalization

• URL: http://arxiv.org/abs/2608.19746v1
• Date: Thu, 20 Aug 2026 07:48:20 GMT
• Title: PersonalBench: Measuring the Authorship Gap in LLM Personalization
• Abstract summary: We introduce PersonalBench, a benchmark that evaluates inference-time personalization methods.<n>Across 50 authors, 1,000 generations, and two model families, we find that personalization methods do produce author-differentiated output.<n>All methods achieve LUAR similarity to real authors in the range 0.484-0.508, below the cross-author human floor of 0.626.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Personalized text generation aims to make LLMs write in a specific individual's style, yet existing benchmarks measure task accuracy or preference alignment rather than whether the model's output actually resembles the target author's writing. We introduce PersonalBench, a benchmark that evaluates inference-time personalization methods through three independent lenses: LUAR (a trained authorship verification model), an LLM-as-judge, and automated stylometrics. Across 50 authors, 1,000 generations, and two model families (Qwen 3, GLM-4), we find that personalization methods do produce author-differentiated output (LUAR discriminates target authors within generated text at AUC=0.918) but this differentiation never crosses the human-LLM boundary. All methods achieve LUAR similarity to real authors in the range 0.484-0.508, below the cross-author human floor of 0.626 (ceiling 0.756). The LLM's own authorship fingerprint dominates: generated text is more distant from any human author than random humans are from each other. Methods are statistically indistinguishable from each other on LUAR (spread 0.024) despite appearing differentiated on the LLM judge, a discrepancy we trace to circularity between trait extraction and profile extraction. We validate that LUAR reliably measures authorship in our corpus (AUC=0.76 single-post, 0.96 multi-post). We release PersonalBench as a calibrated measuring stick: inference-time personalization modulates the LLM's style but does not bridge the gap to human authorship.

Related papers

• Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization [0.0]
  We show that grounding evaluation in authorship verification theory transforms what benchmarks can measure.<n>We evaluate four inference-time personalization methods across 50 authors and 1,000 generations.
  arXiv  Detail & Related papers  (2026-04-29T09:17:01Z)
• Can LLMs Predict Academic Collaboration? Topology Heuristics vs. LLM-Based Link Prediction on Real Co-authorship Networks [4.675202949988315]
  We study whether large language models (LLMs) can predict future scientific collaborations using only author profiles, without access to graph structure.<n>Using Qwen-722.5B-Instruct across three historical eras of AI research, we find that LLMs and topologys capture distinct signals and are strongest in complementary settings.
  arXiv  Detail & Related papers  (2026-04-01T20:39:12Z)
• Can We Hide Machines in the Crowd? Quantifying Equivalence in LLM-in-the-loop Annotation Tasks [8.246529401043128]
  We aim to explore how labeling decisions -- by both humans and LLMs -- can be statistically evaluated across individuals.<n>We develop a statistical evaluation method based on Krippendorff's $alpha$, paired bootstrapping, and the Two One-Sided t-Tests (TOST) equivalence test procedure.<n>We apply this approach to two datasets -- MovieLens 100K and PolitiFact -- and find that the LLM is statistically indistinguishable from a human annotator in the former.
  arXiv  Detail & Related papers  (2025-10-08T05:17:33Z)
• How Well Do LLMs Imitate Human Writing Style? [2.3754840025365183]
  Large language models (LLMs) can generate fluent text, but their ability to replicate the distinctive style of a specific human author remains unclear.<n>We present a fast, training-free framework for authorship verification and style imitation analysis.<n>It achieves 97.5% accuracy on academic essays and 94.5% in cross-domain evaluation.
  arXiv  Detail & Related papers  (2025-09-29T15:34:40Z)
• Evaluating Style-Personalized Text Generation: Challenges and Directions [13.84471733325089]
  Style personalization is highly specific, relative to every user, and depends strongly on the pragmatic context.<n>We examine the effectiveness of the most common metrics used in the field, such as BLEU, embeddings, and LLMs-as-judges.<n>We find strong evidence that employing ensembles of diverse evaluation metrics consistently outperforms single-evaluator methods.
  arXiv  Detail & Related papers  (2025-08-08T15:07:31Z)
• A Bayesian Approach to Harnessing the Power of LLMs in Authorship Attribution [57.309390098903]
  Authorship attribution aims to identify the origin or author of a document. Large Language Models (LLMs) with their deep reasoning capabilities and ability to maintain long-range textual associations offer a promising alternative. Our results on the IMDb and blog datasets show an impressive 85% accuracy in one-shot authorship classification across ten authors.
  arXiv  Detail & Related papers  (2024-10-29T04:14:23Z)
• DALD: Improving Logits-based Detector without Logits from Black-box LLMs [56.234109491884126]
  Large Language Models (LLMs) have revolutionized text generation, producing outputs that closely mimic human writing. We present Distribution-Aligned LLMs Detection (DALD), an innovative framework that redefines the state-of-the-art performance in black-box text detection. DALD is designed to align the surrogate model's distribution with that of unknown target LLMs, ensuring enhanced detection capability and resilience against rapid model iterations.
  arXiv  Detail & Related papers  (2024-06-07T19:38:05Z)
• Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore [51.65730053591696]
  We propose a simple yet effective black-box zero-shot detection approach based on the observation that human-written texts typically contain more grammatical errors than LLM-generated texts.<n> Experimental results show that our method outperforms current state-of-the-art (SOTA) zero-shot and supervised methods.
  arXiv  Detail & Related papers  (2024-05-07T12:57:01Z)
• Beyond Traditional Benchmarks: Analyzing Behaviors of Open LLMs on Data-to-Text Generation [0.0]
  We analyze the behaviors of open large language models (LLMs) on the task of data-to-text (D2T) generation. We find that open LLMs can generate fluent and coherent texts in zero-shot settings from data in common formats collected with Quintd.
  arXiv  Detail & Related papers  (2024-01-18T18:15:46Z)
• Generating Benchmarks for Factuality Evaluation of Language Models [61.69950787311278]
  We propose FACTOR: Factual Assessment via Corpus TransfORmation, a scalable approach for evaluating LM factuality. FACTOR automatically transforms a factual corpus of interest into a benchmark evaluating an LM's propensity to generate true facts from the corpus vs. similar but incorrect statements. We show that: (i) our benchmark scores increase with model size and improve when the LM is augmented with retrieval; (ii) benchmark score and perplexity do not always agree on model ranking; (iii) when perplexity and benchmark score disagree, the latter better reflects factuality in open-ended generation.
  arXiv  Detail & Related papers  (2023-07-13T17:14:38Z)
• Statistical Knowledge Assessment for Large Language Models [79.07989821512128]
  Given varying prompts regarding a factoid question, can a large language model (LLM) reliably generate factually correct answers? We propose KaRR, a statistical approach to assess factual knowledge for LLMs. Our results reveal that the knowledge in LLMs with the same backbone architecture adheres to the scaling law, while tuning on instruction-following data sometimes compromises the model's capability to generate factually correct text reliably.
  arXiv  Detail & Related papers  (2023-05-17T18:54:37Z)
• Self-Refine: Iterative Refinement with Self-Feedback [62.78755306241981]
  Self-Refine is an approach for improving initial outputs from large language models (LLMs) through iterative feedback and refinement. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.
  arXiv  Detail & Related papers  (2023-03-30T18:30:01Z)
• PART: Pre-trained Authorship Representation Transformer [52.623051272843426]
  Authors writing documents imprint identifying information within their texts.<n>Previous works use hand-crafted features or classification tasks to train their authorship models.<n>We propose a contrastively trained model fit to learn textbfauthorship embeddings instead of semantics.
  arXiv  Detail & Related papers  (2022-09-30T11:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.