Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning

• URL: http://arxiv.org/abs/2603.15981v1
• Date: Mon, 16 Mar 2026 22:54:12 GMT
• Title: Aligning Paralinguistic Understanding and Generation in Speech LLMs via Multi-Task Reinforcement Learning
• Abstract summary: Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds.<n>We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning.<n>Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models.
• Score: 40.8328755967818
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds--crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio) by 8-12% on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs.

Related papers

• Resurfacing Paralinguistic Awareness in Large Audio Language Models [64.48666302711108]
  Large Audio Language Models (LALMs) have expanded the interaction with human to speech modality.<n>LALMs usually neglect such paralinguistic cues and respond solely based on query content.<n>We propose a paralinguistic-enhanced fine-tuning (PE-FT) protocol accordingly to equip LALMs with paralinguistic-aware capabilities.
  arXiv  Detail & Related papers  (2026-03-12T13:56:42Z)
• Equipping LLM with Directional Multi-Talker Speech Understanding Capabilities [20.51281468416298]
  We propose two novel approaches to integrate directivity into large language models (LLM)<n>All of the approaches utilize a multi-microphone array embedded in smart glasses to optimize directivity interpretation and processing in a streaming manner.
  arXiv  Detail & Related papers  (2026-02-06T21:45:48Z)
• EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognition [0.0]
  Emotion recognition from speech is a challenging task that requires capturing both linguistic and paralinguistic cues.<n>Recent works have highlighted the ability of Large Language Models (LLMs) to perform tasks outside of the sole natural language area.<n>This work proposes a novel approach that fine-tunes an LLM with audio and text representations for emotion prediction.
  arXiv  Detail & Related papers  (2025-08-19T06:58:16Z)
• Dual Information Speech Language Models for Emotional Conversations [48.094826104102204]
  Speech-language models (SLMs), which use speech as input, are emerging as a promising solution.<n>We identify entangled information and improper training strategies as key issues.<n>Our approach disentangles paralinguistic and linguistic information, enabling SLMs to interpret speech through structured representations.
  arXiv  Detail & Related papers  (2025-08-11T15:33:44Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• Self-Powered LLM Modality Expansion for Large Speech-Text Models [62.27700381806554]
  Large language models (LLMs) exhibit remarkable performance across diverse tasks. This study aims to refine the use of speech datasets for LSM training by addressing the limitations of vanilla instruction tuning. We introduce a self-powered LSM that leverages augmented automatic speech recognition data generated by the model itself for more effective instruction tuning.
  arXiv  Detail & Related papers  (2024-10-04T04:34:24Z)
• DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Data [84.01401439030265]
  Recent end-to-end speech language models (SLMs) have expanded upon the capabilities of large language models (LLMs)<n>We present a simple yet effective automatic process for creating speech-text pair data.<n>Our model demonstrates general capabilities for speech-related tasks without the need for speech instruction-tuning data.
  arXiv  Detail & Related papers  (2024-09-30T07:01:21Z)
• Beyond Silent Letters: Amplifying LLMs in Emotion Recognition with Vocal Nuances [3.396456345114466]
  We propose SpeechCueLLM, a method that translates speech characteristics into natural language descriptions.<n>We evaluate SpeechCueLLM on two datasets: IEMOCAP and MELD, showing significant improvements in emotion recognition accuracy.
  arXiv  Detail & Related papers  (2024-07-31T03:53:14Z)
• SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding [61.02342238771685]
  Spoken language understanding requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. Various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. We propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules.
  arXiv  Detail & Related papers  (2020-10-05T19:29:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.