GSRM: Generative Speech Reward Model for Speech RLHF

• URL: http://arxiv.org/abs/2602.13891v1
• Date: Sat, 14 Feb 2026 21:22:55 GMT
• Title: GSRM: Generative Speech Reward Model for Speech RLHF
• Abstract summary: Generative Speech Reward Model (GSRM) is a reasoning-centric reward model tailored for speech.<n>The GSRM is trained to decompose speech naturalness evaluation into an interpretable acoustic feature extraction stage.<n> Experiments show that GSRM substantially outperforms existing speech naturalness predictors.
• Score: 28.19230481779836
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in speech language models, such as GPT-4o Voice Mode and Gemini Live, have demonstrated promising speech generation capabilities. Nevertheless, the aesthetic naturalness of the synthesized audio still lags behind that of human speech. Enhancing generation quality requires a reliable evaluator of speech naturalness. However, existing naturalness evaluators typically regress raw audio to scalar scores, offering limited interpretability of the evaluation and moreover fail to generalize to speech across different taxonomies. Inspired by recent advances in generative reward modeling, we propose the Generative Speech Reward Model (GSRM), a reasoning-centric reward model tailored for speech. The GSRM is trained to decompose speech naturalness evaluation into an interpretable acoustic feature extraction stage followed by feature-grounded chain-of-thought reasoning, enabling explainable judgments. To achieve this, we curated a large-scale human feedback dataset comprising 31k expert ratings and an out-of-domain benchmark of real-world user-assistant speech interactions. Experiments show that GSRM substantially outperforms existing speech naturalness predictors, achieving model-human correlation of naturalness score prediction that approaches human inter-rater consistency. We further show how GSRM can improve the naturalness of speech LLM generations by serving as an effective verifier for online RLHF.

Related papers

• Speech2Grasp: Data-Efficient Transfer of Text-Conditioned Grasp Detection to Speech in Humanoid Robots [6.597552492273294]
  Speech2Grasp is a framework for data-efficient transfer of text-conditioned grasp detection to speech.<n>Our findings suggest a practical paradigm for extending established text-conditioned systems to speech.
  arXiv  Detail & Related papers  (2026-07-29T07:40:00Z)
• SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models [9.873637206435967]
  Streaming speech-to-speech language models aim to answer spoken queries directly with synthetic speech.<n>Standard benchmarks do not capture whether these systems behave naturally in conversations.<n>We introduce SPEARBench, a benchmark for evaluating naturalness in speech-to-speech language models from question-answer interactions.
  arXiv  Detail & Related papers  (2026-07-06T17:42:59Z)
• On the Emotion Understanding of Synthesized Speech [63.13411068766772]
  Emotion is a core paralinguistic feature in voice interaction.<n>Current Speech Emotion Recognition (SER) models can not generalize to synthesized speech.<n>generative Speech Language Models (SLMs) tend to infer emotion from textual semantics while ignoring paralinguistic cues.
  arXiv  Detail & Related papers  (2026-03-17T13:11:14Z)
• Reward Modeling from Natural Language Human Feedback [77.75758630455357]
  Reinforcement Learning with Verifiable reward (RLVR) on preference data has become the mainstream approach for training Generative Reward Models (GRMs)<n>In this paper, we demonstrate that such binary classification tasks make GRMs susceptible to guessing correct outcomes without sound critiques.<n>We propose Reward Modeling from Natural Language Human Feedback (RM-NLHF), which leverages natural language feedback to obtain process reward signals.
  arXiv  Detail & Related papers  (2026-01-12T09:23:43Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• Speech-DRAME: A Framework for Human-Aligned Benchmarks in Speech Role-Play [68.54773980519457]
  Speech-DRAME is a unified framework that contributes at three levels.<n>It provides the first comprehensive, reproducible foundation for assessing spoken role-play.
  arXiv  Detail & Related papers  (2025-11-03T06:12:40Z)
• SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models [60.72029578488467]
  SpeechR is a unified benchmark for evaluating reasoning over speech in large audio-language models.<n>It evaluates models along three key dimensions: factual retrieval, procedural inference, and normative judgment.<n> Evaluations on eleven state-of-the-art LALMs reveal that high transcription accuracy does not translate into strong reasoning capabilities.
  arXiv  Detail & Related papers  (2025-08-04T03:28:04Z)
• SpeechAlign: Aligning Speech Generation to Human Preferences [51.684183257809075]
  We introduce SpeechAlign, an iterative self-improvement strategy that aligns speech language models to human preferences. We show that SpeechAlign can bridge the distribution gap and facilitate continuous self-improvement of the speech language model.
  arXiv  Detail & Related papers  (2024-04-08T15:21:17Z)
• Evaluating Speech Synthesis by Training Recognizers on Synthetic Speech [34.8899247119748]
  We propose an evaluation technique involving the training of an ASR model on synthetic speech and assessing its performance on real speech. Our proposed metric demonstrates a strong correlation with both MOS naturalness and MOS intelligibility when compared to SpeechLMScore and MOSNet.
  arXiv  Detail & Related papers  (2023-10-01T15:52:48Z)
• SpeechLMScore: Evaluating speech generation using speech language model [43.20067175503602]
  We propose SpeechLMScore, an unsupervised metric to evaluate generated speech using a speech-language model. It does not require human annotation and is a highly scalable framework. Evaluation results demonstrate that the proposed metric shows a promising correlation with human evaluation scores on different speech generation tasks.
  arXiv  Detail & Related papers  (2022-12-08T21:00:15Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• Evaluating Models of Robust Word Recognition with Serial Reproduction [8.17947290421835]
  We compare several broad-coverage probabilistic generative language models in their ability to capture human linguistic expectations. We find that those models that make use of abstract representations of preceding linguistic context best predict the changes made by people in the course of serial reproduction.
  arXiv  Detail & Related papers  (2021-01-24T20:16:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.