Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set: A Common-Reference Staged Normalization Analysis

• URL: http://arxiv.org/abs/2608.16379v1
• Date: Mon, 17 Aug 2026 10:31:30 GMT
• Title: Unadapted Multilingual ASR on a Garrusi Kurdish Evaluation Set: A Common-Reference Staged Normalization Analysis
• Abstract summary: I evaluate speech recognition for a Kurdish variety written in a Latin field orthography.<n>The raw Arabic-script hypothesis scores 111.70% WER and 100.92% CER, with zero exact word matches.<n>Latin transliteration gives 102.36% WER and 57.89% CER; folding it into the reference's reduced orthography gives 97.85% and 51.20%.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating speech recognition for a Kurdish variety written in a Latin field orthography, using a model that outputs Arabic script, creates a measurement problem before a modelling one: direct scoring treats writing-system differences as recognition errors. Jointly normalizing reference and hypothesis avoids this, but also changes reference tokenization, mixing agreement gains with a change in the scoring denominator. I evaluate MMS-1B-all with the Central Kurdish (ckb) adapter, used as released without adaptation, on 1,722 Garrusi questionnaire segments from five speakers (9,763 reference word tokens; 117.9 minutes). I use a common-reference design: the reference is folded once and fixed at 9,763 tokens, while only the hypothesis representation varies. The raw Arabic-script hypothesis scores 111.70% WER and 100.92% CER, with zero exact word matches. Latin transliteration gives 102.36% WER and 57.89% CER; folding it into the reference's reduced orthography gives 97.85% and 51.20%. Thus RAW-to-FOLDED reduces measured WER by 13.85 points and CER by 49.72 points; folding alone accounts for 4.51 and 6.69 points. Substantial error remains: 14.53% of reference tokens are exact matches, edits are substitution-dominated, and per-segment WER is higher for shorter segments. A Southern Kurdish fine-tuned system (aranemini/southern-kurdish-asr), scored under the same design, performs worse on every speaker (1,703 segments), with 109.56% WER and 55.85% CER. However, 12,330 output characters fall outside the folding table, so these rates must be recomputed against the corrected fixed reference. The MMS output also contains 613 unconverted or unmapped characters, showing that part of the residual error reflects scoring-pipeline limits rather than recognition alone. I will release the fixed reference and segment-level results, subject to source-corpus sharing terms, to support independent checking.

Related papers

• PA-CDM: Position-Aware Character Detection Matching for Evaluating Handwritten Mathematical Expression Recognition [0.0]
  Handwritten mathematical expression recognition is scored by exact-match rates and string-similarity metrics that are blind to where an error occurs.<n>We propose a position-aware metric that couples character detection matching with position-forest encoding and divergence-level weighting.<n>In a six-annotator study, PA-CDM attains the highest correlation with human judgments among seven automatic metrics.
  arXiv  Detail & Related papers  (2026-09-11T14:43:05Z)
• Can Scene Text Recognition Read Rare Compositions? [1.7844382164707184]
  Scene text recognition is reported as 89--97% accurate on the six standard benchmarks, and the problem is widely treated as saturated.<n>We present an alternative reading.<n>When the same test images are stratified jointly by ground-truth word rarity and character n-gram novelty against a reference corpus, accuracy at the rareword x rare-trigram corner of the resulting 5x5 grid drops 10--18 pt below the q3/q3 centre.
  arXiv  Detail & Related papers  (2026-09-01T07:16:42Z)
• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Stoicheia: Character-Level Masked Diffusion for Ancient Greek Textual Restoration, Parsing, and Metrical Scansion [0.0]
  Stoicheia is a character-level masked-diffusion encoder for Ancient Greek.<n>We pretrain it on an open, revision-pinned corpus of 380M words and release eleven checkpoints.<n>On Ithaca's own test split, with identical frozen samples and strict scoring, Stoicheia reduces character error relative to both prior state-of-the-art systems.
  arXiv  Detail & Related papers  (2026-08-07T14:07:43Z)
• Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps [9.710464466895521]
  We show that a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries.<n>On HumanEval+/MBPP+, a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries.
  arXiv  Detail & Related papers  (2026-08-07T08:14:46Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• One Ruler: A Same-Hands Re-Evaluation of Bivariate Causal Direction on Tuebingen, with a Parameter-Free Compression Baseline [0.0]
  We re-evaluate the Tuebingen cause-effect pairs in which every method is run by us on the identical 102 pairs.<n>As a clean reference point we introduce a deliberately minimal baseline.<n>Under the common ruler the ranking differs sharply from the literature.
  arXiv  Detail & Related papers  (2026-06-22T15:43:19Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• FormInv: A Measurement Protocol for Semantic Invariance in Mathematical Reasoning Benchmarks [0.0]
  A paraphrase-quality audit of MathCheck detected 4 semantically incorrect paraphrases in 129 groups.<n>Removing them drops GPT-4o from rank 2 to rank 4 and elevates Claude Haiku and DeepSeek V3 above it.
  arXiv  Detail & Related papers  (2026-05-27T18:59:18Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Step-wise Rubric Rewards for LLM Reasoning [72.17879367869503]
  Reinforcement Learning with Verifiable Rewards (RLVR) is widely used to improve reasoning in large language models.<n>We find 18.2% of steps in correct responses are wrong yet positively rewarded.<n>Across six mathematical reasoning benchmarks, SRaR improves average accuracy over RaR by 3.57 points.
  arXiv  Detail & Related papers  (2026-05-17T07:08:14Z)
• The Digital Afterlife of Empires: Four Language Models Converge on the Same Imperial Cartography of Writing [0.0]
  Large language models process the world's writing systems with radical inequality.<n>Only 29 scripts (9.7%) are fully supported by contemporary digital infrastructure; among 158 living scripts, 60 (38.0%) lack complete support.<n> Tokenizer efficiency varies by a factor of 31.7 across 45 scripts measured with parallel text.
  arXiv  Detail & Related papers  (2026-04-30T07:02:45Z)
• Error Correction by Paying Attention to Both Acoustic and Confidence References for Automatic Speech Recognition [52.624909026294105]
  We propose a non-autoregressive speech error correction method. A Confidence Module measures the uncertainty of each word of the N-best ASR hypotheses. The proposed system reduces the error rate by 21% compared with the ASR model.
  arXiv  Detail & Related papers  (2024-06-29T17:56:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.