The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora

• URL: http://arxiv.org/abs/2608.09093v1
• Date: Mon, 10 Aug 2026 03:43:30 GMT
• Title: The Announcement Carries the Cue: Markup, Boundaries, and the Notation of Pre-Training Corpora
• Abstract summary: How a document's arrangement is written down, its notation, is a training variable that no dataset card records.<n>We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How a document's arrangement is written down, its notation, is a training variable that no dataset card records. The field has established that text-extraction choices change model behaviour, and has never once measured the notation of what those choices put into the corpus. We define clean-window survival, a deterministic count of how much of a stream still demands the boundary inference, and measure notation on three fronts. What corpora carry: a census of thirteen public corpora, where survival falls to 0.153 in a vision-converted PDF slice against 0.889 in C4; the scarce resource is not unmarked text but long unmarked text; a pre-registered supply test finds what remains institutional, not consumer. Our own pre-registered prediction failed: converters do not fabricate structure on prose, and that null forced the reliability mechanism that survives it. What readers use: across five base models spanning 0.6B to 8.2B and two pipelines, deleting a structural announcement makes the following prose measurably harder to predict, while swapping its notation moves nothing. That zero does not make notation unimportant; it relocates the variable: the operative cue is the announcement, not the sigil. What writers impose: a bounded null. Base models do not impose the marked register above the authored baseline, and handed prose with every announcement deleted they do not put one back, at a rate indistinguishable from zero against an authored reference of zero. We ship the format those measurements imply: the pure frame, paragraphs in authored order, every announcement deleted into a reversible sidecar, mixed against the marked copy over announcement presence rather than notation. Choose format operators by the capability they train, not by the fidelity they preserve, and record extractor identity and survival on data cards.

Related papers

• A Memorization Floor for LLM Refinement of Decompiled Code [0.0]
  We introduce a within-item control separating what LLM output recovers from its input from what it recovers from its prior.<n>Because the comparison is within-item, corpus difficulty cannot contribute; it costs twenty API calls.<n> refiner from another vendor reproduces this -- twelve contrasts, two models, twelve nulls.<n>No registered hypothesis was confirmed, and we report the five instrument failures behind that in full.
  arXiv  Detail & Related papers  (2026-09-15T14:15:08Z)
• RelateAnything: Real-Time Open-Vocabulary Relation Prediction From Any Inputs [0.40611352512781873]
  RelateAnything takes an image and regions from any source and returning scored relations over a predicate vocabulary supplied at inference as strings.<n>Training over 19,103 predicates requires positive-unlabeled supervision and a text encoder that separates antonyms.<n>On three cross-dataset benchmarks and a fourth zero-shot, RelateAnything has 2.3-3.5x the mean recall of the strongest open-vocabulary method of comparable scale.
  arXiv  Detail & Related papers  (2026-09-11T07:57:41Z)
• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints [4.41537705949485]
  Language-model judges now gate training data, score generations, and drive leaderboards.<n>We audited that assumption in two preregistered campaigns with every threshold fixed in advance.<n>Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.<n>We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist.
  arXiv  Detail & Related papers  (2026-09-03T17:59:43Z)
• VeriPhy: Agentic Physical Reasoning for World Model Evaluation and Refinement [57.86962208273888]
  We present an auditable physical-verification system in which a text-only planner compiles the prompt into typed physical obligations.<n>Each action returns a provenance-carrying evidence record whose payload, when usable, is either a typed measurement or an explicitly tagged learned state.<n>We anchor evaluation in a 1,500-clip corpus of human-annotated flaw records that localize real generation failures in prompt reference, space, and time.
  arXiv  Detail & Related papers  (2026-09-02T20:36:45Z)
• A Glyph Is Not a Letter, a Token Is Not a Word, a Space Is Not a Space: What the Units of Voynichese Are Not [45.88028371034407]
  We analyse the Voynich manuscript (Beinecke MS 408) with matched prose, cipher, and pseudo-text controls and quire-level resampling.<n>None holds, and the failures share a shape: the order in Voynichese sits at the edges of tokens and at graded boundaries between them.<n>A published Voynich-imitating cipher and a self-citation text generator both reproduce the low entropy, the unit scale, the weak token order, and the null result of a calibrated substitution attack.
  arXiv  Detail & Related papers  (2026-08-17T20:05:12Z)
• Metaphor Tracer: A Theory-Informed Analysis of Hidden States [0.0]
  From one forward pass, without training, we score every token position on two properties.<n>The *aggregator* measures whether the position consolidates the whole text into a stable configuration.<n>The *differentiator*, whether other tokens are transiently carried into its subspace as the model reads.
  arXiv  Detail & Related papers  (2026-07-30T16:12:13Z)
• Measuring Alignment With Reader Highlights Net of Position and Length [1.8620637029128544]
  On 120 web documents (at least 12 independent readers each), a language-model importance ranking keeps 38.4% of crowd-marked sentences against 19.9% of their matched neighbours.<n>A single human reader reaches +0.182 - indistinguishable from GPT-5.4 (+0.002 [-0.081, +0.088]) and below 5.
  arXiv  Detail & Related papers  (2026-07-30T06:24:38Z)
• Prompt Design at Scale: How Format, Instruction Count, and Context Length Shape Instruction Adherence and Hallucination in Large Language Models [0.0]
  Two experiments cross all three factors on one held, contamination-free synthetic corpus.<n>Experiment 1 measures instruction-following decay as rule count N grows from 10 to 160, crossed with four formats and system-prompt vs. user-turn placement.<n>Experiment 2 measures recall accuracy, false-premise sycophancy, and absent-fact fabrication across a 2k-to-512k-token context ladder.
  arXiv  Detail & Related papers  (2026-07-21T16:31:35Z)
• Abliteration Is Not a Scalpel: Off-Target Effects of Refusal Removal on Decision Disposition Across Model Families [51.56484100374058]
  Abliteration - deleting a model's refusal direction from its weights - is the standard recipe behind popular "uncensored" open-weight models.<n>As a disposition probe we use 21,600 decisions under uncertainty, replayed through a frozen pipeline so the decision-layer model is the only variable.<n>Three effects replicate across both families (weeks-clustered CIs excluding zero)<n>A fourth effect reverses sign: the same operation makes Gemma-abliterated less confident and Qwen-abliterated more (family CIs non-overlapping)
  arXiv  Detail & Related papers  (2026-07-19T22:27:00Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• Legible-by-Construction: Attention and End-to-End Transformers [0.2663471820643486]
  A transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations.<n> hidden units read as named logical operators at no cost to language-model quality.<n>Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become legible.
  arXiv  Detail & Related papers  (2026-07-05T14:08:59Z)
• Automated High-Precision Extraction and Forensic Verification of Data-Bearing Vector Figures [45.39546965981896]
  The quantitative record of science and engineering is increasingly carried by figures rather than text or tables.<n>When a figure is stored as graphics, its data are not approximated by the picture but encoded in it.<n>We turn this into three contributions, one per shortcoming of hand digitization.
  arXiv  Detail & Related papers  (2026-06-30T08:43:32Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks [47.027290803102666]
  We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation.<n>A pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark.
  arXiv  Detail & Related papers  (2026-06-02T13:41:43Z)
• Not All Synthetic Data Is Yours to Learn From [68.13136636413601]
  We show that weak self-training can amplify capabilities already present in the pretrained model.<n>We study this in the minimal setting of prompt-free unconditional self-training.
  arXiv  Detail & Related papers  (2026-05-29T10:34:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.