Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes

• URL: http://arxiv.org/abs/2608.23640v1
• Date: Sun, 23 Aug 2026 20:10:50 GMT
• Title: Auditing the Synthetic Memoir: Measuring Scene-Level Confabulation in LLM-Generated Autobiography Against the Documented Record of the Life It Describes
• Abstract summary: We present the first quantified audit of a large language model's autobiography against a subject-specific ground-truth corpus.<n>A 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM.<n>Every day was audited at the anecdote-scene level against an independent verification corpus.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When a large language model (LLM) is asked to write a person's life, how much of what it writes actually happened? We present a scene-level case-study audit - the first quantified audit of LLM-generated autobiography against a subject-specific ground-truth corpus that we are aware of, based on an unsystematic literature search. The subject and the author of this paper are the same person: a 366-day "page-a-day" book of first-person anecdotal entries was drafted with a conversational LLM whose documented inputs were a template, two exemplar days, and each day's quote - not her corpus - and every day was subsequently audited at the anecdote-scene level against an independent verification corpus using a four-level rubric fixed before analysis. We define the verification-failure rate as the share of days not rated VERIFIED (scene positively corroborated): 354 of 366 days fail, 96.7% (Wilson 95% CI 94.4-98.1%). Only 12 days contain a corroborated scene; 19 days (5.2%) assert claims actively contradicted by the record; the dominant failure mode is grounded drift - real people, employers, and settings inside invented scenes - though its measured share varies across raters. Independent re-rating replicates the headline (no evidence the original rate was inflated) while showing that the four-way taxonomy has only fair-to-moderate reliability. Regenerating the same days with current named models reproduces 100% verification failure under the same inputs; grounding generation in the subject's corpus significantly improves the verification rate while leaving substantial residual failure (83.3%). We contribute the measurement, a reusable audit instrument whose WEAK/UNVERIFIED boundary we show to be unreliable, and a grounding remedy with quantified effect.

Related papers

• NovGauge: A Fine-Grained Benchmark for Diagnosing LLMs' Capability in Paper Novelty Assessment [54.4265627247104]
  Large language models (LLMs) are increasingly used in peer review at major AI conferences.<n>Existing benchmarks assess novelty as a single holistic score.<n>We present NovGauge, a human-anchored benchmark for fine-grained novelty assessment diagnosis.
  arXiv  Detail & Related papers  (2026-09-10T08:34:56Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• Benchmark Contamination: A Taxonomy Organized by Defeated Mitigation [0.0]
  A benchmark score is a joint property of the model, the evaluation harness, the elicitation budget, the sampled population, and contamination status.<n>We introduce a taxonomy organized by the mitigation each type defeats.<n>Two variables fall below the prevalence-robust threshold registered in advance.
  arXiv  Detail & Related papers  (2026-08-29T23:04:09Z)
• Reconcile Once, Write Anytime: A Trust-Tiered Librarian and a Multi-Agent Writer for Drift-Free, Point-in-Time Research [7.426963307811556]
  We present a two-tier agentic system that separates a maintained, point-in-time knowledge library from report writing.<n>A portable multi-agent "writer" runtime composes a contradiction-free, evidence-grounded report at any knowledge cutoff T.<n>We evaluate on a self-collected, public corpus of 6,130 sources yielding 555,926 evidence cards.
  arXiv  Detail & Related papers  (2026-08-13T09:09:28Z)
• The Half-Lives of Generative-AI Evidence: A 40-Record Audit, a Claim-Currency Framework, and a Reflexive Case of Frontier-Model-Assisted Research [0.0]
  Generative-AI evaluations can become historical before publication, yet calendar age does not affect every conclusion equally.<n>This paper audits a maximum-variation purposive corpus of 40 empirical records appearing between 18 July 2025 and 17 July 2026.<n>The audit coded publication route, execution timing, model identity, age of the newest named generation or immutable snapshot, same-family supersession and refresh behaviour.
  arXiv  Detail & Related papers  (2026-07-27T05:59:49Z)
• Plato-Bio: verification-first biological novelty screening with temporal rediscovery and structural benchmarks [0.0]
  We developed Plato-Bio, a biology-routed extension of the open Plato/Denario architecture.<n>Plato-Bio couples explicit workflow states with provenance records, citation checks, claim-to-evidence links, scoped file writes, and publication gates.<n>On the current clean revision, the full Python suite completed with 931 passes, six skips, and no failures or errors.
  arXiv  Detail & Related papers  (2026-07-27T03:55:39Z)
• Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings [0.0]
  A seeded life-script sampler emits facts with validity intervals volatility classes, and source channels before any text exists.<n>A verifier confirms every planted fact and questions are instantiated mechanically from the script.<n>The synthetic, fictionalized corpus embeds features absent from the benchmarks we survey.
  arXiv  Detail & Related papers  (2026-07-24T04:19:45Z)
• The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol [0.0]
  Studies of bias in LLM-as-judge systems typically build synthetic corpora by prompting an LLM to generate a hallucinated answer to pair with a factual one, then presenting both to a judge.<n>We report a case in which this generation step silently failed, and use it to argue that the failure mode is structural rather than incidental.<n>A positive control supports this claim directly: an analogous fault injected into a minimal perturbation-based corpus is caught with 100% accuracy by a zero-cost, zero-human gold-to-negative string comparison.
  arXiv  Detail & Related papers  (2026-07-15T11:12:54Z)
• HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam [63.84155758655084]
  Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models.<n>We introduce HLE-Verified, a verified and revised version of HLE with a transparent verification protocol and fine-grained error taxonomy.<n>We evaluate seven state-of-the-art language models on HLE and HLE-Verified, observing an average absolute accuracy gain of 7--10 percentage points.
  arXiv  Detail & Related papers  (2026-02-15T02:50:15Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• Investigating Factuality in Long-Form Text Generation: The Roles of Self-Known and Self-Unknown [68.33486915047014]
  We investigate the factuality of long-form text generation across various large language models (LLMs)<n>Our analysis reveals that factuality tend to decline in later sentences of the generated text, accompanied by a rise in the number of unsupported claims.
  arXiv  Detail & Related papers  (2024-11-24T22:06:26Z)
• LLMs Can Patch Up Missing Relevance Judgments in Evaluation [56.51461892988846]
  We use large language models (LLMs) to automatically label unjudged documents. We simulate scenarios with varying degrees of holes by randomly dropping relevant documents from the relevance judgment in TREC DL tracks. Our method achieves a Kendall tau correlation of 0.87 and 0.92 on an average for Vicuna-7B and GPT-3.5 Turbo respectively.
  arXiv  Detail & Related papers  (2024-05-08T00:32:19Z)
• FABLES: Evaluating faithfulness and content selection in book-length summarization [55.50680057160788]
  In this paper, we conduct the first large-scale human evaluation of faithfulness and content selection on book-length documents. We collect FABLES, a dataset of annotations on 3,158 claims made in LLM-generated summaries of 26 books, at a cost of $5.2K USD. An analysis of the annotations reveals that most unfaithful claims relate to events and character states, and they generally require indirect reasoning over the narrative to invalidate.
  arXiv  Detail & Related papers  (2024-04-01T17:33:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.