Prediction of User Request and Complaint in Spoken Customer-Agent Conversations

• URL: http://arxiv.org/abs/2208.10249v1
• Date: Wed, 27 Jul 2022 21:29:38 GMT
• Title: Prediction of User Request and Complaint in Spoken Customer-Agent Conversations
• Abstract summary: This corpus includes a transcription of the spoken conversations and was divided into two sets: Train and Devel sets. We have investigated 14 feature sets: 6 linguistic feature sets, 6 audio feature sets and 2 vocal interaction feature sets. The results show that the linguistic features always give the best results (91.2% for the Request task and 70.3% for the Complaint task)
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present the corpus called HealthCall. This was recorded in real-life conditions in the call center of Malakoff Humanis. It includes two separate audio channels, the first one for the customer and the second one for the agent. Each conversation was anonymized respecting the General Data Protection Regulation. This corpus includes a transcription of the spoken conversations and was divided into two sets: Train and Devel sets. Two important customer relationship management tasks were assessed on the HealthCall corpus: Automatic prediction of type of user requests and complaints detection. For this purpose, we have investigated 14 feature sets: 6 linguistic feature sets, 6 audio feature sets and 2 vocal interaction feature sets. We have used Bidirectional Encoder Representation from Transformers models for the linguistic features, openSMILE and Wav2Vec 2.0 for the audio features. The vocal interaction feature sets were designed and developed from Turn Takings. The results show that the linguistic features always give the best results (91.2% for the Request task and 70.3% for the Complaint task). The Wav2Vec 2.0 features seem more suitable for these two tasks than the ComPaRe16 features. Vocal interaction features outperformed ComPaRe16 features on Complaint task with a 57% rate achieved with only six features.

Related papers

• Continue, Adapt, or Yield: In-Turn Adaptation to Overlapping Speech in Full-Duplex Agents [2.745780834143671]
  We introduce Duplex Cue, an evaluation of thisemphin adaptation in full voice agents.<n>In a single-model study using 300 human-confirmed unscripted English conversations, we compare recorded human responses generated while replaying listener's cues.<n>Speakers adapt in 68.2% of cases, compared with 32.4% for PersonaPlex.
  arXiv  Detail & Related papers  (2026-09-11T17:46:10Z)
• MP-Bench: Evaluating Voice Agents as a Multiparty Conversation Participant [138.90198190949545]
  We introduce MP-Bench, the first benchmark specifically designed to evaluate conversational speech systems as active participants within multi-party contexts.<n>MP-Bench assesses agent behavior along two primary dimensions: turn-taking awareness and response appropriateness.<n>We find that real-time voice agents stay at or below 22% on multiparty comprehension and remain near chance on multiparty turn-taking.
  arXiv  Detail & Related papers  (2026-09-11T17:11:55Z)
• SteerDuplex: Steerable Duplex Speech Dialogue Models [47.277615720741046]
  Full spoken dialogue model fine-tuned on natural conversations and synthetic dialogues targeting following instruction, vocal delivery, reasoning, and interaction.<n>We apply two-stage reinforcement learning (RL) with hybrid rewards, combining verifiable interaction checks and judge-based feedback to improve response continuity.<n> RL further raises source-clean interruption response from 72.5% to 82.5% and reduces synthetic pause barge-in from 26.5% to 9%.
  arXiv  Detail & Related papers  (2026-09-11T09:22:46Z)
• Multimodal Duplex Interaction Agent [48.78411737142625]
  Gander is an interaction model that processes streaming user inputs, enabling full interaction in both everyday conversations and complex scenarios.<n>We evaluate Gander across conversational ability, interactive capability, understanding, and tool assisted task execution.<n>Gander supports a range of challenging interaction settings, including background noise interference, multi-party interactions, and backchannel communication.
  arXiv  Detail & Related papers  (2026-09-08T16:22:23Z)
• Leading-Silence Augmentation and Multi-Stage Synthetic Supervision for the Second MLC-SLM Challenge [28.363948904202932]
  The second Multilingual Conversational Speech Language Model (MLC-SLM) Challenge evaluates two tasks over complete, unsegmented multilingual conversations.<n>Neither task provides oracle utterance boundaries or speaker labels at evaluation, and Task 2 provides no question-answer training set.<n>We fine-tune VibeVoice-ASR-7B with random leading-silence cropping, consistent timestamp correction, and distribution-matched augmentation.
  arXiv  Detail & Related papers  (2026-08-14T10:00:42Z)
• VoiceAgentBench: Are Voice Assistants ready for agentic tasks? [5.639970295197759]
  We introduce VoiceAgentBench, a benchmark to evaluate SpeechLMs in realistic spoken agentic settings.<n>It comprises over 5,500 synthetic spoken queries grounded in Indian context.<n>It measures tool selection accuracy, structural consistency, and the correctness of tool invocations, including adversarial robustness.
  arXiv  Detail & Related papers  (2025-10-09T09:11:38Z)
• AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation [55.607230723223346]
  This work presents a systematic study of Large Audio Model (LAM) as a Judge, AudioJudge, investigating whether it can provide a unified evaluation framework that addresses both challenges.<n>We explore AudioJudge across audio characteristic detection tasks, including pronunciation, speaking rate, speaker identification and speech quality, and system-level human preference simulation for automated benchmarking.<n>We introduce a multi-aspect ensemble AudioJudge to enable general-purpose multi-aspect audio evaluation. This method decomposes speech assessment into specialized judges for lexical content, speech quality, and paralinguistic features, achieving up to 0.91 Spearman correlation with human preferences on
  arXiv  Detail & Related papers  (2025-07-17T00:39:18Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• WavReward: Spoken Dialogue Models With Generalist Reward Evaluators [57.80264359636158]
  WavReward is a reward feedback model that can evaluate IQ and EQ of spoken dialogue systems with speech input.<n>ChatReward-30K is a preference dataset used to train WavReward.<n>WavReward outperforms previous state-of-the-art evaluation models across multiple spoken dialogue scenarios.
  arXiv  Detail & Related papers  (2025-05-14T16:54:15Z)
• Multi-Domain Audio Question Answering Toward Acoustic Content Reasoning in The DCASE 2025 Challenge [102.84031769492708]
  This task defines three QA subsets to test audio-language models on interactive question-answering over diverse acoustic scenes.<n>Preliminary results on the development set are compared, showing strong variation across models and subsets.<n>This challenge aims to advance the audio understanding and reasoning capabilities of audio-language models toward human-level acuity.
  arXiv  Detail & Related papers  (2025-05-12T09:04:16Z)
• Target conversation extraction: Source separation using turn-taking dynamics [23.189364779538757]
  We introduce the novel task of target conversation extraction, where the goal is to extract the audio of a target conversation based on the speaker embedding of one of its participants. Using neural networks, we show the feasibility of our approach on English and Mandarin conversation datasets. In the presence of interfering speakers, our results show an 8.19 dB improvement in signal-to-noise ratio for 2-speaker conversations and a 7.92 dB improvement for 2-4-speaker conversations.
  arXiv  Detail & Related papers  (2024-07-15T22:55:27Z)
• AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension [95.8442896569132]
  We introduce AIR-Bench, the first benchmark to evaluate the ability of Large Audio-Language Models (LALMs) to understand various types of audio signals and interact with humans in the textual format. Results demonstrate a high level of consistency between GPT-4-based evaluation and human evaluation.
  arXiv  Detail & Related papers  (2024-02-12T15:41:22Z)
• Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue [71.15186328127409]
  Paralinguistics-enhanced Generative Pretrained Transformer (ParalinGPT) Model takes the conversational context of text, speech embeddings, and paralinguistic attributes as input prompts within a serialized multitasking framework. We utilize the Switchboard-1 corpus, including its sentiment labels as the paralinguistic attribute, as our spoken dialogue dataset.
  arXiv  Detail & Related papers  (2023-12-23T18:14:56Z)
• Question-Interlocutor Scope Realized Graph Modeling over Key Utterances for Dialogue Reading Comprehension [61.55950233402972]
  We propose a new key utterances extracting method for dialogue reading comprehension. It performs prediction on the unit formed by several contiguous utterances, which can realize more answer-contained utterances. As a graph constructed on the text of utterances, we then propose Question-Interlocutor Scope Realized Graph (QuISG) modeling.
  arXiv  Detail & Related papers  (2022-10-26T04:00:42Z)
• Real-time Caller Intent Detection In Human-Human Customer Support Spoken Conversations [10.312382727352823]
  Agent assistance during human-human customer support spoken interactions requires triggering based on the caller's intent (reason for call) The goal is for a system to detect the caller's intent at the time the agent would have been able to detect it (Intent Boundary) Recent work on voice assistants has used incremental real-time predictions at a word-by-word level to detect intent before the end of a command.
  arXiv  Detail & Related papers  (2022-08-14T07:50:23Z)
• Multi-task Voice-Activated Framework using Self-supervised Learning [0.9864260997723973]
  Self-supervised learning methods such as wav2vec 2.0 have shown promising results in learning speech representations from unlabelled and untranscribed speech data. We propose a general purpose framework for adapting a pre-trained wav2vec 2.0 model for different voice-activated tasks.
  arXiv  Detail & Related papers  (2021-10-03T19:28:57Z)
• WASE: Learning When to Attend for Speaker Extraction in Cocktail Party Environments [21.4128321045702]
  In the speaker extraction problem, it is found that additional information from the target speaker contributes to the tracking and extraction of the speaker. Inspired by the cue of sound onset, we explicitly modeled the onset cue and verified the effectiveness in the speaker extraction task. From the perspective of tasks, our onset/offset-based model completes the composite task, a complementary combination of speaker extraction and speaker-dependent voice activity detection.
  arXiv  Detail & Related papers  (2021-06-13T14:56:05Z)
• Dialogue History Matters! Personalized Response Selectionin Multi-turn Retrieval-based Chatbots [62.295373408415365]
  We propose a personalized hybrid matching network (PHMN) for context-response matching. Our contributions are two-fold: 1) our model extracts personalized wording behaviors from user-specific dialogue history as extra matching information. We evaluate our model on two large datasets with user identification, i.e., personalized dialogue Corpus Ubuntu (P- Ubuntu) and personalized Weibo dataset (P-Weibo)
  arXiv  Detail & Related papers  (2021-03-17T09:42:11Z)
• AdaSpeech: Adaptive Text to Speech for Custom Voice [104.69219752194863]
  We propose AdaSpeech, an adaptive TTS system for high-quality and efficient customization of new voices. Experiment results show that AdaSpeech achieves much better adaptation quality than baseline methods, with only about 5K specific parameters for each speaker.
  arXiv  Detail & Related papers  (2021-03-01T13:28:59Z)
• FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention [66.77490220410249]
  We propose FragmentVC, in which the latent phonetic structure of the utterance from the source speaker is obtained from Wav2Vec 2.0. FragmentVC is able to extract fine-grained voice fragments from the target speaker utterance(s) and fuse them into the desired utterance. This approach is trained with reconstruction loss only without any disentanglement considerations between content and speaker information.
  arXiv  Detail & Related papers  (2020-10-27T09:21:03Z)
• HarperValleyBank: A Domain-Specific Spoken Dialog Corpus [7.331287001215395]
  HarperValleyBank is a free, public domain spoken dialog corpus. The data simulate simple consumer banking interactions, containing about 23 hours of audio from 1,446 human-human conversations.
  arXiv  Detail & Related papers  (2020-10-26T22:16:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.