TimeStampEval: A Simple LLM Eval and a Little Fuzzy Matching Trick to Improve Search Accuracy

• URL: http://arxiv.org/abs/2511.11594v1
• Date: Mon, 27 Oct 2025 21:54:56 GMT
• Title: TimeStampEval: A Simple LLM Eval and a Little Fuzzy Matching Trick to Improve Search Accuracy
• Abstract summary: TimeStampEval is a benchmark for retrieving precise millisecond timestamps from long transcripts given non-verbatim quotes.<n>Our simple two-stage method dramatically improves retrieval accuracy while cutting inference costs by over 90%.<n>The motivating use case is an automated long-form podcast that assembles Congressional Record clips into AI-hosted verbatim narration.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional fuzzy matching often fails when searching for quotes that are semantically identical but syntactically different across documents-a common issue when aligning official written records with speech-to-text transcripts. We introduce TimeStampEval, a benchmark for retrieving precise millisecond timestamps from long transcripts given non-verbatim quotes. Our simple two-stage method dramatically improves retrieval accuracy while cutting inference costs by over 90%. The motivating use case is an automated long-form podcast that assembles Congressional Record clips into AI-hosted narration. The technical challenge: given a sentence-timestamped transcript and a target quote that may differ due to transcription or editorial drift, return exact start and end boundaries. Standard algorithms handle verbatim text but break under fuzzier variants. Evaluating six modern LLMs on a 2,800-sentence (120k-token) transcript revealed four key findings. (1) Prompt design matters more than model choice: placing the query before the transcript and using compact formatting improved accuracy by 3-20 points while reducing token count by 30-40%. (2) Off-by-one errors form a distinct category, showing models understand the task but misplace boundaries. (3) A modest reasoning budget (600-850 tokens) raises accuracy from 37% to 77% for weak setups and to above 90% for strong ones. (4) Our "Assisted Fuzzy" approach-RapidFuzz pre-filtering followed by LLM verification on short snippets-improves fuzzy match accuracy by up to 50 points while halving latency and reducing cost per correct result by up to 96%. Extended tests on ten transcripts (50k-900k tokens, 1989-2025) confirm robustness to transcript length, vocabulary drift, and domain change, maintaining 95-100% rejection accuracy for absent targets.

Related papers

• Transcription Policy as a Latent Variable: Activating Controllable Verbatim ASR with Word-Level Timing [8.921166277011347]
  We show that models already encode both styles; the challenge is controlled activation.<n>Using coverage-aware decoder task tokens trained on parallel verbatim/intended transcript pairs, we raise German disfluency F1 from 10% to 79% zero-shot.<n>Full English-only fine-tuning surpasses all baselines in verbatim accuracy, disfluency detection, and intended-mode quality across both languages.
  arXiv  Detail & Related papers  (2026-07-21T10:19:17Z)
• REDDIT: Correcting Model-Generated Timestamp Drift in ASR without Forgetting via Replay-Based Distribution Editing [46.75114860324604]
  Autoregressive ASR systems can emit timestamps as decoded tokens, enabling timestamped transcription without frame-level or inference-time post-processing.<n>We show that these generated timestamps can drift across long non-speech spans.<n>We propose REDDIT(REplay-based Distribution eDITing), a lightweight two-stage post-training framework that corrects timestamps while avoiding forgetting.
  arXiv  Detail & Related papers  (2026-07-06T17:40:54Z)
• The Script Tax: Measuring Tokenization-Driven Efficiency and Latency Disparities in Multilingual Language Models [0.0]
  We quantify a script tax by comparing two orthographic variants with identical linguistic content.<n>Across mBERT and XLM-R, the higher-fragmentation orthography shows a 3.4x increase in fertility.<n>Using bits per character (BPC) to avoid the "NLL paradox" from subword fragmentation, we find a substantial increase in information cost.
  arXiv  Detail & Related papers  (2026-01-19T14:45:40Z)
• Distinguishing Repetition Disfluency from Morphological Reduplication in Bangla ASR Transcripts: A Novel Corpus and Benchmarking Analysis [0.0]
  We introduce the first publicly available, 20,000-row Bangla corpus, manually annotated to explicitly distinguish between these two phenomena in noisy ASR transcripts.<n>We benchmark this novel resource using two paradigms: state-of-the-art multilingual Large Language Models (LLMs) and task-specific fine-tuning of encoder models.
  arXiv  Detail & Related papers  (2025-11-17T09:06:01Z)
• Prompt-Based One-Shot Exact Length-Controlled Generation with LLMs [56.47577824219207]
  We present a prompt-based strategy that compels an off-the-shelf large language model to generate exactly a desired number of tokens.<n>The prompt appends countdown markers and explicit counting rules so that the model "writes while counting"<n>On MT-Bench-LI, strict length compliance with GPT-4.1 leaps from below 30% under naive prompts to above 95% with our countdown prompt.
  arXiv  Detail & Related papers  (2025-08-19T13:12:01Z)
• Small Edits, Big Consequences: Telling Good from Bad Robustness in Large Language Models [0.0]
  Large language models (LLMs) now write code in settings where misreading a single word can break safety or cost money, yet we still expect them to overlook stray typos.<n>To probe where useful robustness ends and harmful insensitivity begins, we compile 50 LeetCode problems and craft three minimal prompt perturbations.<n>Six frontier models, including three "reasoning-tuned" versions, solve each mutated prompt.
  arXiv  Detail & Related papers  (2025-07-15T03:22:07Z)
• Predicting Compact Phrasal Rewrites with Large Language Models for ASR Post Editing [18.962260162806988]
  Large Language Models (LLMs) excel at rewriting tasks such as text style transfer and grammatical error correction.<n>We propose alternative edit phrase representations inspired by phrase-based statistical machine translation.
  arXiv  Detail & Related papers  (2025-01-23T16:54:27Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [74.11403803488643]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation.<n>We show that QASemConsistency yields factual consistency scores that correlate well with human judgments.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Augmenting Automatic Speech Recognition Models with Disfluency Detection [12.45703869323415]
  Speech disfluency commonly occurs in conversational and spontaneous speech. Current research mainly focuses on detecting disfluencies within transcripts, overlooking their exact location and duration in the speech. We present an inference-only approach to augment any ASR model with the ability to detect open-set disfluencies.
  arXiv  Detail & Related papers  (2024-09-16T11:13:14Z)
• Handling Numeric Expressions in Automatic Speech Recognition [56.972851337263755]
  We compare cascaded and end-to-end approaches to recognize and format numeric expressions.<n>Results show that adapted end-to-end models offer competitive performance with the advantage of lower latency and inference cost.
  arXiv  Detail & Related papers  (2024-07-18T09:46:19Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• SpellMapper: A non-autoregressive neural spellchecker for ASR customization with candidate retrieval based on n-gram mappings [76.87664008338317]
  Contextual spelling correction models are an alternative to shallow fusion to improve automatic speech recognition. We propose a novel algorithm for candidate retrieval based on misspelled n-gram mappings. Experiments on Spoken Wikipedia show 21.4% word error rate improvement compared to a baseline ASR system.
  arXiv  Detail & Related papers  (2023-06-04T10:00:12Z)
• On Addressing Practical Challenges for RNN-Transduce [72.72132048437751]
  We adapt a well-trained RNN-T model to a new domain without collecting the audio data. We obtain word-level confidence scores by utilizing several types of features calculated during decoding. The proposed time stamping method can get less than 50ms word timing difference on average.
  arXiv  Detail & Related papers  (2021-04-27T23:31:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.