A Unified and Reproducible Experimentation Framework for Speech Understanding

• URL: http://arxiv.org/abs/2605.30899v1
• Date: Fri, 29 May 2026 06:33:36 GMT
• Title: A Unified and Reproducible Experimentation Framework for Speech Understanding
• Abstract summary: SURE is a unified experimentation framework that standardizes prediction formats, normalization, and scoring.<n>We present SURE, a unified experimentation framework that standardizes prediction formats, normalization, and scoring.
• Score: 44.73703262229875
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech foundation models and Speech LLMs have advanced speech understanding, yet deployment-oriented model selection is hindered by non-comparable evaluations caused by mismatched post-processing, and by training results that are hard to reproduce across data scales and pipelines. We present SURE, a unified experimentation framework that standardizes prediction formats, normalization, and scoring. SURE evaluates strong systems across paradigms, from conventional pipelines to Speech LLMs, on representative tasks under realistic acoustic and linguistic stressors. Beyond evaluation, SURE introduces an agent-assisted training conversion flow that maps paper and code into versioned, runnable training pipelines under a unified protocol on matched open-data subsets. Overall, SURE improves comparability and reproducibility for deployment-oriented evaluation.

Related papers

• PARM: Pipeline-Adapted Reward Model [60.769414637325326]
  Reward models (RMs) are central to aligning large language models (LLMs) with human preferences, powering advanced decoding strategies.<n>While most prior work focuses on single-step generation, real-world applications increasingly adopt multi-stage pipelines.<n>We investigate this through code generation for optimization, constructing a pipeline that integrates reward models into both formulation and solution stages.
  arXiv  Detail & Related papers  (2026-04-20T14:29:08Z)
• A Dialectic Pipeline for Improving LLM Robustness [0.0]
  Methods such as fine-tuning on domain-specific data or the training of a separate textitad hoc verifier require demanding computational resources.<n>We propose a dialectic pipeline that preserves LLMs' generalization abilities while improving the quality of its answer via self-dialogue.<n>We find that our proposed dialectic pipeline is able to outperform by significative margins the standard model answers.
  arXiv  Detail & Related papers  (2026-01-28T14:42:49Z)
• Refinement Provenance Inference: Detecting LLM-Refined Training Prompts from Model Behavior [58.751981587234916]
  This paper formalizes the Refinement Provenance Inference (RPI) audit task as Refinement Provenance Inference (RPI)<n>We propose RePro, a logit-based framework that fuses teacher-forced likelihood features with logit-ranking signals.<n>During training, RePro learns a transferable representation via shadow fine-tuning, and uses a lightweight linear head to infer provenance on unseen victims without training-data access.
  arXiv  Detail & Related papers  (2026-01-05T10:16:41Z)
• AURORA: Augmented Understanding via Structured Reasoning and Reinforcement Learning for Reference Audio-Visual Segmentation [113.75682363364004]
  AURORA is a framework designed to enhance genuine reasoning and language comprehension in reference audio-visual segmentation.<n>AURORA achieves state-of-the-art performance on Ref-AVS benchmarks and generalizes effectively to unreferenced segmentation.
  arXiv  Detail & Related papers  (2025-08-04T07:47:38Z)
• A Novel Data Augmentation Approach for Automatic Speaking Assessment on Opinion Expressions [8.717610965852037]
  We propose a novel training paradigm to generate diverse responses of a given proficiency level.<n>We convert responses into synthesized speech via speaker-aware text-to-speech synthesis.<n>A multimodal large language model integrates aligned textual features with speech signals to predict proficiency scores directly.
  arXiv  Detail & Related papers  (2025-06-04T15:42:53Z)
• Improving Generalization of Speech Separation in Real-World Scenarios: Strategies in Simulation, Optimization, and Evaluation [45.76004686788507]
  We present a novel data simulation pipeline that produces diverse training data from a range of acoustic environments and content. We propose new training paradigms to improve quality of a general speech separation model.
  arXiv  Detail & Related papers  (2024-08-28T20:26:34Z)
• Policy-Gradient Training of Language Models for Ranking [29.940468096858066]
  Text retrieval plays a crucial role in incorporating factual knowledge for decision making into language processing pipelines. Current state-of-the-art text retrieval models leverage pre-trained large language models (LLMs) to achieve competitive performance. We introduce Neural PG-RANK, a novel training algorithm that learns to rank by instantiating a LLM as a Plackett-Luce ranking policy.
  arXiv  Detail & Related papers  (2023-10-06T17:55:23Z)
• How to Estimate Model Transferability of Pre-Trained Speech Models? [84.11085139766108]
  "Score-based assessment" framework for estimating transferability of pre-trained speech models. We leverage upon two representation theories, Bayesian likelihood estimation and optimal transport, to generate rank scores for the PSM candidates. Our framework efficiently computes transferability scores without actual fine-tuning of candidate models or layers.
  arXiv  Detail & Related papers  (2023-06-01T04:52:26Z)
• An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition [98.70304981174748]
  We focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
  arXiv  Detail & Related papers  (2021-10-09T15:06:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.