Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback

• URL: http://arxiv.org/abs/2601.19063v1
• Date: Tue, 27 Jan 2026 00:55:14 GMT
• Title: Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback
• Abstract summary: We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models.<n>Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness.
• Score: 82.70507055599093
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning from human or AI feedback (RLHF/RLAIF) for speech-in/speech-out dialogue systems (SDS) remains underexplored, with prior work largely limited to single semantic rewards applied at the utterance level. Such setups overlook the multi-dimensional and multi-modal nature of conversational quality, which encompasses semantic coherence, audio naturalness, speaker consistency, emotion alignment, and turn-taking behavior. Moreover, they are fundamentally mismatched with duplex spoken dialogue systems that generate responses incrementally, where agents must make decisions based on partial utterances. We address these limitations with the first multi-reward RLAIF framework for SDS, combining semantic, audio-quality, and emotion-consistency rewards. To align utterance-level preferences with incremental, blockwise decoding in duplex models, we apply turn-level preference sampling and aggregate per-block log-probabilities within a single DPO objective. We present the first systematic study of preference learning for improving SDS quality in both multi-turn Chain-of-Thought and blockwise duplex models, and release a multi-reward DPO dataset to support reproducible research. Experiments show that single-reward RLAIF selectively improves its targeted metric, while joint multi-reward training yields consistent gains across semantic quality and audio naturalness. These results highlight the importance of holistic, multi-reward alignment for practical conversational SDS.

Related papers

• Multi-Faceted Interactivity Alignment in Full-Duplex Speech Models [53.470209949659115]
  We propose a post-training alignment method that improves full spoken dialogue models through RL.<n>We address the four canonical axes of interactivity: handling turn-taking, back-channel pause, and user interruption.<n>An extra-based reward for response quality prevents semantic degradation.
  arXiv  Detail & Related papers  (2026-06-09T17:46:55Z)
• WavAlign: Enhancing Intelligence and Expressiveness in Spoken Dialogue Models via Adaptive Hybrid Post-Training [42.85483422955689]
  We propose a modality-aware adaptive post-training recipe for spoken dialogue.<n>It constrains preference updates to the semantic channel and improves acoustic behavior via explicit anchoring.<n>We evaluate the method across multiple spoken dialogue benchmarks and representative architectures.
  arXiv  Detail & Related papers  (2026-04-16T12:20:27Z)
• Dual-Axis Generative Reward Model Toward Semantic and Turn-taking Robustness in Interactive Spoken Dialogue Models [45.119381322968735]
  Well-designed reward signals are crucial for reinforcement learning (RL)<n>Our model achieves state-of-the-art performance on interaction-quality assessment across a wide spectrum of datasets.
  arXiv  Detail & Related papers  (2026-04-16T12:03:50Z)
• Interactive ASR: Towards Human-Like Interaction and Semantic Coherence Evaluation for Agentic Speech Recognition [52.63676763985825]
  We propose a semantic-aware evaluation metric to assess recognition quality beyond token-level accuracy.<n>We design an agent framework to simulate human-like multi-turn interaction, enabling iterative refinement of recognition outputs.<n>We will release the code to facilitate future research in interactive and agentic ASR.
  arXiv  Detail & Related papers  (2026-04-10T09:02:42Z)
• Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics [99.89493037369071]
  Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts.<n>Recent LLM-based systems perform well on single-speaker benchmarks, but their robustness in multi-speaker settings is unclear.<n>We systematically compare LLM-based and modular approaches along four axes: overlap, semantic fidelity, speaker count, and single- versus multi-channel input.
  arXiv  Detail & Related papers  (2026-03-24T02:01:21Z)
• Discourse-Aware Dual-Track Streaming Response for Low-Latency Spoken Dialogue Systems [31.911085541071028]
  We propose a low-latency architecture that enables listen-while-thinking and speak-while-thinking.<n>Experiments on two spoken dialogue benchmarks demonstrate that DDTSR reduces response latency by 19%-51%.
  arXiv  Detail & Related papers  (2026-02-26T17:39:56Z)
• Covo-Audio Technical Report [61.09708870154148]
  Covo-Audio, a 7B-end LALM, directly processes continuous audio inputs and generates audio outputs within a single unified architecture.<n>Covo-Audio-Chat, a dialogue-oriented variant, demonstrates semantic strong spoken conversational abilities.
  arXiv  Detail & Related papers  (2026-02-10T14:31:11Z)
• Multi-granularity Interactive Attention Framework for Residual Hierarchical Pronunciation Assessment [18.97451964522765]
  We propose a novel residual hierarchical interactive method, HIA, that enables bidirectional modeling across granularities.<n>We also propose a residual hierarchical structure to alleviate the feature forgetting problem when modeling acoustic hierarchies.<n>Our model is comprehensively ahead of the existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-01-05T02:43:04Z)
• High-Fidelity Speech Enhancement via Discrete Audio Tokens [35.61634772862795]
  DAC-SE1 is a language model-based SE framework leveraging discrete high-resolution audio representations.<n>Our experiments show that DAC-SE1 surpasses state-of-the-art autoregressive SE methods on both objective perceptual metrics and in a MUSHRA human evaluation.
  arXiv  Detail & Related papers  (2025-10-02T16:38:05Z)
• Session-Level Spoken Language Assessment with a Multimodal Foundation Model via Multi-Target Learning [8.717610965852037]
  Spoken Language Assessment (SLA) estimates a learner's oral proficiency from spontaneous speech.<n>This paper introduces a novel multimodal foundation model approach that performs session-level evaluation in a single pass.
  arXiv  Detail & Related papers  (2025-09-19T14:33:05Z)
• Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks [81.44256822500257]
  RLHF has emerged as a predominant approach for aligning artificial intelligence systems with human preferences.<n> RLHF exhibits insufficient compliance capabilities when confronted with complex multi-instruction tasks.<n>We propose a novel Multi-level Aware Preference Learning (MAPL) framework, capable of enhancing multi-instruction capabilities.
  arXiv  Detail & Related papers  (2025-05-19T08:33:11Z)
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities [93.09944267871163]
  FullDuplexBench is a benchmark that systematically evaluates key interactive behaviors.<n>By releasing our benchmark code we aim to advance spoken dialogue modeling and the development of more natural and engaging SDMs.
  arXiv  Detail & Related papers  (2025-03-06T18:59:16Z)
• Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder [58.523884148942166]
  We propose two novel techniques to improve audio-visual speech recognition (AVSR) under a pre-training and fine-tuning training framework. First, we explore the correlation between lip shapes and syllable-level subword units in Mandarin to establish good frame-level syllable boundaries from lip shapes. Next, we propose an audio-guided cross-modal fusion encoder (CMFE) neural network to utilize main training parameters for multiple cross-modal attention layers.
  arXiv  Detail & Related papers  (2023-08-14T08:19:24Z)
• Unified Modeling of Multi-Talker Overlapped Speech Recognition and Diarization with a Sidecar Separator [42.8787280791491]
  Multi-talker overlapped speech poses a significant challenge for speech recognition and diarization. We propose a cost-effective method to convert a single-talker automatic speech recognition system into a multi-talker one. We incorporate a diarization branch into the Sidecar, allowing for unified modeling of both ASR and diarization with a negligible overhead of only 768 parameters.
  arXiv  Detail & Related papers  (2023-05-25T17:18:37Z)
• Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System [49.39150449455407]
  HDNO is an option framework for designing latent dialogue acts to avoid designing specific dialogue act representations. We test HDNO on MultiWoz 2.0 and MultiWoz 2.1, the datasets on multi-domain dialogues, in comparison with word-level E2E model trained with RL, LaRL and HDSA.
  arXiv  Detail & Related papers  (2020-06-11T20:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.