No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus

• URL: http://arxiv.org/abs/2608.28875v1
• Date: Fri, 28 Aug 2026 21:26:35 GMT
• Title: No Detectable Change in Side-Level WER from Prompt-Level Context: A Preregistered Ablation on a Production Oral-History Corpus
• Abstract summary: Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain.<n>This work tested that mechanism, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Supplying context at inference time to a large multimodal model is an inexpensive lever for adapting speech transcription to a domain, and earlier results on smaller models reported large gains. This work tested that mechanism where it ships, in the prompt-conditioning layer of a production oral-history transcription tool, on a sample from its own production corpus. Full prompt-level context did not detectably change side-level word error rate (WER), and none of the four preregistered hypotheses was supported. The design was a within-item paired ablation, preregistered with the analysis code frozen by hash before the confirmatory batch was scored; two disclosed gpt-4o pilot sides had been scored earlier, during scorer development. Nineteen cassette sides, about 10.6 hours of degraded 1970s-80s interview audio, were reprocessed through the production code path under three prompt arms, crossed with two deployed commercial configurations, gpt-4o-transcribe and gemini-2.5-flash, and scored against operator-corrected verbatim references. For gpt-4o-transcribe the median paired difference between the full-context and no-context arms was +0.6 WER points, with a side-resampled interval of [-1.1, +1.0]; the Gemini estimates were too unstable to support a comparable negative inference. A post-hoc rerun found run-to-run pipeline variability larger than the confirmatory differences, so effects of that size cannot be resolved from one transcription per cell. An implementation audit verified the manipulation was live, and sequence-alignment analysis found a small improvement on complete context-listed phrases, too small to materially change side-level WER, and for Gemini coexisting with worsened unlisted-token error. Evaluating context mechanisms therefore requires sequence-aligned term-level, insertion, and speaker-label measures alongside aggregate accuracy.

Related papers

• Prediction of Prediction (PoP): Inter-Layer Activation Fusion for Single-Pass Hallucination Detection in Large Language Models [0.0]
  This work evaluates whether internal hidden-state transition dynamics during generation can signal factual errors without auxiliary decoding calls.<n>We introduce Prediction of Prediction (PoP), a mechanism that captures layer-transition uncertainty.<n>PoP achieves an area under the receiver operating characteristic curve (AUROC) of 75.5% for factual-correctness classification.
  arXiv  Detail & Related papers  (2026-08-27T14:17:14Z)
• Prior Audit-Repair Context Shifts LLM Verifier Thresholds Toward Leniency [0.025718125188898048]
  We measure false alarms on human-verified-correct ProcessBench traces with the present task held byte-identical.<n>We find that a completed audit -> repair episode already in the model's context lowers false alarms in 15 of 15 model x wording combinations.
  arXiv  Detail & Related papers  (2026-08-17T01:41:43Z)
• Making two action heads agree: coordination mechanisms and a runtime collapse certificate for flow-matching policies [13.05465000489116]
  Flow-matching policy decodes each predicted motion into joint and end-effector spaces.<n>On multimodal tasks, independently sampled branches may choose different valid modes, causing false alarms.<n>We study how to coordinate the two branches and at what cost.
  arXiv  Detail & Related papers  (2026-08-16T13:57:36Z)
• Independent Patch Verification for Coding Agents with a Bidirectional Reconstruct-and-Verify Framework [49.16055123488827]
  coding agents powered by large language models can now generate code patches directly from bug reports.<n>No mechanism independently verifies whether it truly resolves the reported problem.<n>We propose RETRACE, a training-free post-generation verification framework.
  arXiv  Detail & Related papers  (2026-08-09T22:59:13Z)
• Trajectories That Segment Themselves: Agent-Declared Boundaries as a Training Unit [0.0]
  We introduce collection-time semantic self-segmentation, in which a declarative contract has the acting agent expose its own boundaries while the trajectory is generated.<n>Because the agent names its conjecture, a reviewer can negate it by name, which lets our protocol manufacture wrong-cause-then-correction transitions that recorded work rarely contains.<n>We then ask what survives deleting the declaration. Given the cut points but not the hypothesis, a model attributes action blocks to their governing hypothesis at over twice chance.<n>Asked instead to place boundaries, a code-blind annotator matches 24 of 40 where random placement matches 11.5, while a mechanical test
  arXiv  Detail & Related papers  (2026-08-03T14:27:58Z)
• Hacking Generative Perplexity: Why Unconditional Text Evaluation Needs Distributional Metrics [49.443264461057645]
  Diffusion and continuous flow-based language models have emerged as the leading non-autoregressive alternatives to language modeling.<n>Progress in both paradigms is overwhelmingly tracked by generative perplexity (gen-PPL): the per-token negative log-likelihood of samples under a frozen autoregressive (AR) scorer.<n>We argue that this metric is unsound. By construction, gen-PPL measures only predictability under the scoring AR, not grammaticality or semantic coherence.
  arXiv  Detail & Related papers  (2026-06-07T02:35:56Z)
• A Held-Out Transition-Pair Falsifier for Long-Horizon Non-Abelian State Tracking [0.5076419064097734]
  We introduce a held-out transition-pair falsifier for finite non-Abelian group tracking.<n>In a controlled $S_3 times S_3$ benchmark, a projected recurrent state model trained only on length-8 sequences produces error-free final-state predictions.
  arXiv  Detail & Related papers  (2026-06-05T13:26:41Z)
• ExComm: Exploration-Stage Communication for Error-Resilient Agentic Test-Time Scaling [57.42714978834704]
  ExComm is a communication protocol for exploration-stage agentic test-time scaling.<n>We show that ExComm consistently outperforms strong test-time scaling baselines.
  arXiv  Detail & Related papers  (2026-05-21T07:38:44Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• End-to-End Page-Level Assessment of Handwritten Text Recognition [69.55992406968495]
  HTR systems increasingly face the end-to-end page-level transcription of a document. Standard metrics do not take into account the inconsistencies that might appear. We propose a two-fold evaluation, where the transcription accuracy and the RO goodness are considered separately.
  arXiv  Detail & Related papers  (2023-01-14T15:43:07Z)
• Factual Error Correction for Abstractive Summaries Using Entity Retrieval [57.01193722520597]
  We propose an efficient factual error correction system RFEC based on entities retrieval post-editing process. RFEC retrieves the evidence sentences from the original document by comparing the sentences with the target summary. Next, RFEC detects the entity-level errors in the summaries by considering the evidence sentences and substitutes the wrong entities with the accurate entities from the evidence sentences.
  arXiv  Detail & Related papers  (2022-04-18T11:35:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.