What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts

• URL: http://arxiv.org/abs/2609.12085v1
• Date: Thu, 10 Sep 2026 18:11:41 GMT
• Title: What Counts as a Mistake? Annotating Recitation Events in Quran Memorization Transcripts
• Abstract summary: We report a completed human annotation of 100 production recording cases.<n>In a preliminary pilot, eight single 20-minute runs across three coding agents and eight models span label-aware F1 0.143 to 0.892.<n>Seven of 162 events defeat all six same-day runs, five of them one orthographic rule, and the strongest run still misses the same ones.
• Score: 2.627150003429427
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Checking Quran recitation from an ASR transcript requires distinguishing unresolved mistakes from repetitions, repairs, opening formulas and accepted spelling differences. We report a completed human annotation of 100 production recording cases: 348 scored units and 162 localized events across ten combined labels. An executable evaluator scores labels and word positions together. A plain diff reaches label-aware F1 0.525 and localization F1 0.826; adapted production cleaner/alignment components reach 0.518 and 0.786, with exact-span F1 0.505 for both. Correcting the adapter's word coordinates recovers all five annotated repetition events, showing why annotation interfaces must be checked before interpreting baseline failures. In a preliminary pilot, eight single 20-minute runs across three coding agents and eight models span label-aware F1 0.143 to 0.892: seven land far above every baseline, and one collapses below the naive diff from a missing normalization step. Across the six, 970 of 972 gold-event instances draw an overlapping prediction, so what remains is not detection but convention: span extent, and the labels whose boundary is stipulated by adjudication rather than visible in the text. Seven of 162 events defeat all six same-day runs, five of them one orthographic rule, and the strongest run still misses the same ones. No run annotated before building, so the pilot measures the algorithm half of the task only.

Related papers

• Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints [4.41537705949485]
  Language-model judges now gate training data, score generations, and drive leaderboards.<n>We audited that assumption in two preregistered campaigns with every threshold fixed in advance.<n>Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90.<n>We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist.
  arXiv  Detail & Related papers  (2026-09-03T17:59:43Z)
• Shortcut Before Circuit: Document Statistics Time In-Context Conflict Resolution [0.0]
  We train transformers on a synthetic language where recency and rarity are exactly coextensive.<n>We separate them with a minimal causal edit that inverts one cue while holding the truth, token count and answer position fixed.
  arXiv  Detail & Related papers  (2026-08-25T12:09:19Z)
• Feedback That Backfires: Why Small Language Model Agents Repeat the Call They Just Watched Fail [0.0]
  Defining the corrective gain of a failure record as the change in log-probability of re-emitting the action that just failed, we find the gain is negative for every instruction-tuned model.<n>The problem is in the harness, not the model's grasp of error messages, and that predicts which remedies work.
  arXiv  Detail & Related papers  (2026-08-24T10:19:17Z)
• Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set: A Common-Reference Staged Normalization Analysis [0.0]
  I evaluate speech recognition for a Kurdish variety written in a Latin field orthography.<n>The raw Arabic-script hypothesis scores 111.70% WER and 100.92% CER, with zero exact word matches.<n>Latin transliteration gives 102.36% WER and 57.89% CER; folding it into the reference's reduced orthography gives 97.85% and 51.20%.
  arXiv  Detail & Related papers  (2026-08-17T10:31:30Z)
• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• The Low Frequency Trap: Video Language Models Fail at Simple Event Bookkeeping [70.07961857926377]
  We introduce trace-grounded parametric profiling for event counting in three controlled video tasks.<n>Across 2,190 videos, we vary event count N and frequency F while holding rendering fixed.<n>At an 80% reliability threshold, Gemini 3.6 Flash reliably counts persistent state transitions up to 12 events at 0.5 and 1.0 Hz, yet demonstrates no reliable positive-count region for transient blinking events.
  arXiv  Detail & Related papers  (2026-08-06T17:57:06Z)
• AI_LectureNote: A Retrospective Pilot Study of a Post-ASR Workflow for English-Script Rendering and Semantic Drift in Korean-English Medical Lectures [1.7028841264419619]
  AI_LectureNote is a readability-oriented post-ASR workflow for Korean-English medical lectures.<n>It rewrites speech-to-text output into study transcripts while restoring Latin-script medical terms.
  arXiv  Detail & Related papers  (2026-07-19T13:02:07Z)
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks [47.027290803102666]
  We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation.<n>A pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark.
  arXiv  Detail & Related papers  (2026-06-02T13:41:43Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Right Regions, Wrong Labels: Semantic Label Flips in Segmentation under Correlation Shift [36.96633404761302]
  We show that a model may achieve reasonable overlap while assigning the wrong semantic label.<n>We quantify it with a simple diagnostic (Flip) that counts how often ground truth foreground pixels are assigned the wrong foreground identity.
  arXiv  Detail & Related papers  (2026-04-14T22:15:17Z)
• BibTeX Citation Hallucinations in Scientific Publishing Agents: Evaluation and Mitigation [34.429649156970015]
  Prior evaluations tested base models without search, which does not reflect current practice.<n>Three search-enabled frontier models generate Bib entries scored on nine fields and a six-way error taxonomy.<n>Overall accuracy is 83.6%, but only 50.9% of entries are fully correct.
  arXiv  Detail & Related papers  (2026-04-03T16:30:58Z)
• Semantic F1 Scores: Fair Evaluation Under Fuzzy Class Boundaries [65.89202599399252]
  We propose Semantic F1 Scores, novel evaluation metrics for subjective or fuzzy multi-label classification.<n>By granting partial credit for semantically related but nonidentical labels, Semantic F1 better reflects the realities of domains marked by human disagreement or fuzzy category boundaries.
  arXiv  Detail & Related papers  (2025-09-25T21:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.