Human-Guided Reasoning with Large Language Models for Vietnamese Speech Emotion Recognition

• URL: http://arxiv.org/abs/2604.01711v1
• Date: Thu, 02 Apr 2026 07:24:14 GMT
• Title: Human-Guided Reasoning with Large Language Models for Vietnamese Speech Emotion Recognition
• Abstract summary: This paper proposes a human-machine collaborative framework that integrates human knowledge into the learning process.<n>A confidence-based routing mechanism is introduced to distinguish between easy and ambiguous samples.<n> Experiments are conducted on a Vietnamese speech dataset of 2,764 samples across three emotion classes.
• Score: 1.744935381873858
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vietnamese Speech Emotion Recognition (SER) remains challenging due to ambiguous acoustic patterns and the lack of reliable annotated data, especially in real-world conditions where emotional boundaries are not clearly separable. To address this problem, this paper proposes a human-machine collaborative framework that integrates human knowledge into the learning process rather than relying solely on data-driven models. The proposed framework is centered around LLM-based reasoning, where acoustic feature-based models are used to provide auxiliary signals such as confidence and feature-level evidence. A confidence-based routing mechanism is introduced to distinguish between easy and ambiguous samples, allowing uncertain cases to be delegated to LLMs for deeper reasoning guided by structured rules derived from human annotation behavior. In addition, an iterative refinement strategy is employed to continuously improve system performance through error analysis and rule updates. Experiments are conducted on a Vietnamese speech dataset of 2,764 samples across three emotion classes (calm, angry, panic), with high inter-annotator agreement (Fleiss Kappa = 0.8574), ensuring reliable ground truth. The proposed method achieves strong performance, reaching up to 86.59% accuracy and Macro F1 around 0.85-0.86, demonstrating its effectiveness in handling ambiguous and hard-to-classify cases. Overall, this work highlights the importance of combining data-driven models with human reasoning, providing a robust and model-agnostic approach for speech emotion recognition in low-resource settings.

Related papers

• Team RAS in 11th ABAW Competition: Multimodal Ambivalence Recognition Approach [52.92973028971813]
  We present a single text-centered multimodal approach for video-level ambivalence and hesitancy recognition.<n>The proposed approach combines linguistic, acoustic, facial, and scene features using text-centered multimodal fusion model.<n>Experiments on the Behavioural Ambivalence/Hesitancy (BAH) corpus confirm that text is the strongest unimodal modality.
  arXiv  Detail & Related papers  (2026-07-16T08:04:23Z)
• Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences [8.680081568962999]
  This study investigates the limits of current approaches to learning human preferences in language models.<n>We propose a feature-augmented framework to better capture the multidimensional nature of human judgment.
  arXiv  Detail & Related papers  (2026-04-01T18:26:16Z)
• On the Fallacy of Global Token Perplexity in Spoken Language Model Evaluation [88.77441715819366]
  Generative spoken language models pretrained on large-scale raw audio can continue a speech prompt with appropriate content.<n>We propose a variety of likelihood- and generative-based evaluation methods that serve in place of naive global token perplexity.
  arXiv  Detail & Related papers  (2026-01-09T22:01:56Z)
• REFLEX: Self-Refining Explainable Fact-Checking via Disentangling Truth into Style and Substance [14.932352020762991]
  We propose REason-guided Fact-checking with Latent EXplanations REFLEX paradigm.<n>It is a plug-and-play, self-refining paradigm that leverages the internal knowledge in backbone model to improve both verdict accuracy and explanation quality.<n>With only 465 self-refined training samples, RELFEX achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-11-25T12:06:23Z)
• Large Language Models for Full-Text Methods Assessment: A Case Study on Mediation Analysis [15.98124151893659]
  Large language models (LLMs) offer potential for automating methodological assessments.<n>We benchmarked state-of-the-art LLMs against expert human reviewers across 180 full-text scientific articles.
  arXiv  Detail & Related papers  (2025-10-12T19:04:22Z)
• Data-Efficient ASR Personalization for Non-Normative Speech Using an Uncertainty-Based Phoneme Difficulty Score for Guided Sampling [2.0128859854921743]
  This work introduces a data-efficient personalization method that quantifies phoneme-level uncertainty to guide fine-tuning.<n>We leverage Monte Carlo Dropout to estimate which phonemes a model finds most difficult.<n>Our results show that this clinically-validated, uncertainty-guided sampling significantly improves ASR accuracy, delivering a practical framework for personalized and inclusive ASR.
  arXiv  Detail & Related papers  (2025-09-23T12:54:30Z)
• PAC: Pronunciation-Aware Contextualized Large Language Model-based Automatic Speech Recognition [20.121140251177145]
  This paper addresses two key challenges in Large Language Model (LLM)-based Automatic Speech Recognition (ASR) systems: effective pronunciation modeling and robust homophone discrimination.<n> Experimental results on the public English Librispeech and Mandarin AISHELL-1 datasets indicate that PAC: (1) reduces relative Word Error Rate (WER) by 30.2% and 53.8% compared to pre-trained ASR models, and (2) achieves 31.8% and 60.5% relative reductions in biased WER for long-tail words.
  arXiv  Detail & Related papers  (2025-09-16T04:07:28Z)
• Reinforced Interactive Continual Learning via Real-time Noisy Human Feedback [59.768119380109084]
  This paper introduces an interactive continual learning paradigm where AI models dynamically learn new skills from real-time human feedback.<n>We propose RiCL, a Reinforced interactive Continual Learning framework leveraging Large Language Models (LLMs)<n>Our RiCL approach substantially outperforms existing combinations of state-of-the-art online continual learning and noisy-label learning methods.
  arXiv  Detail & Related papers  (2025-05-15T03:22:03Z)
• HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Models [81.56455625624041]
  We introduce the first open-source benchmark to utilize external large language models (LLMs) for ASR error correction. The proposed benchmark contains a novel dataset, HyPoradise (HP), encompassing more than 334,000 pairs of N-best hypotheses. LLMs with reasonable prompt and its generative capability can even correct those tokens that are missing in N-best list.
  arXiv  Detail & Related papers  (2023-09-27T14:44:10Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)
• Leveraging Pre-trained Language Model for Speech Sentiment Analysis [58.78839114092951]
  We explore the use of pre-trained language models to learn sentiment information of written texts for speech sentiment analysis. We propose a pseudo label-based semi-supervised training strategy using a language model on an end-to-end speech sentiment approach.
  arXiv  Detail & Related papers  (2021-06-11T20:15:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.