GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition

• URL: http://arxiv.org/abs/2405.02151v2
• Date: Sun, 16 Jun 2024 12:35:23 GMT
• Title: GMP-TL: Gender-augmented Multi-scale Pseudo-label Enhanced Transfer Learning for Speech Emotion Recognition
• Authors: Yu Pan, Yuguang Yang, Heng Lu, Lei Ma, Jianjun Zhao,
• Abstract summary: We introduce GMP-TL, a novel SER framework that employs gender-augmented multi-scale pseudo-label (GMP) based transfer learning. We show that GMP-TL attains a WAR of 80.0% and an UAR of 82.0%, achieving superior performance compared to state-of-the-art unimodal SER methods.
• Score: 7.828525577104307
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The continuous evolution of pre-trained speech models has greatly advanced Speech Emotion Recognition (SER). However, current research typically relies on utterance-level emotion labels, inadequately capturing the complexity of emotions within a single utterance. In this paper, we introduce GMP-TL, a novel SER framework that employs gender-augmented multi-scale pseudo-label (GMP) based transfer learning to mitigate this gap. Specifically, GMP-TL initially uses the pre-trained HuBERT, implementing multi-task learning and multi-scale k-means clustering to acquire frame-level GMPs. Subsequently, to fully leverage frame-level GMPs and utterance-level emotion labels, a two-stage model fine-tuning approach is presented to further optimize GMP-TL. Experiments on IEMOCAP show that our GMP-TL attains a WAR of 80.0% and an UAR of 82.0%, achieving superior performance compared to state-of-the-art unimodal SER methods while also yielding comparable results to multimodal SER approaches.

Related papers

• GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning [106.98018881499362]
  We introduce GEPA (Genetic-Pareto), a prompt that thoroughly incorporates natural language to learn high-level rules from trial and error.<n>GEPA samples system-level trajectories (e.g., reasoning, tool calls, and tool outputs) and reflects on them in natural language to diagnose problems.<n>It can often turn even just a few rollouts into a large quality gain.
  arXiv  Detail & Related papers  (2025-07-25T17:42:32Z)
• ProGMLP: A Progressive Framework for GNN-to-MLP Knowledge Distillation with Efficient Trade-offs [42.37669895235534]
  We introduce a Progressive framework designed to offer flexible and on-demand trade-offs between inference cost and accuracy for GNN-to-MLP knowledge.<n>Our approach is validated through comprehensive experiments on eight real-world graph datasets.
  arXiv  Detail & Related papers  (2025-07-25T07:35:09Z)
• UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation [52.12029029338604]
  We introduce UniGen, a unified multimodal large language model (MLLM) capable of image understanding and generation.<n>We study the full training pipeline of UniGen from a data-centric perspective, including multi-stage pre-training, supervised fine-tuning, and direct preference optimization.<n>We propose a new Chain-of-Thought Verification (CoT-V) strategy for test-time scaling, which significantly boosts UniGen's image generation quality.
  arXiv  Detail & Related papers  (2025-05-20T17:59:26Z)
• Leveraging Cross-Attention Transformer and Multi-Feature Fusion for Cross-Linguistic Speech Emotion Recognition [60.58049741496505]
  Speech Emotion Recognition (SER) plays a crucial role in enhancing human-computer interaction. We propose a novel approach HuMP-CAT, which combines HuBERT, MFCC, and prosodic characteristics. We show that, by fine-tuning the source model with a small portion of speech from the target datasets, HuMP-CAT achieves an average accuracy of 78.75%.
  arXiv  Detail & Related papers  (2025-01-06T14:31:25Z)
• Classifier-guided Gradient Modulation for Enhanced Multimodal Learning [50.7008456698935]
  Gradient-Guided Modulation (CGGM) is a novel method to balance multimodal learning with gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS. CGGM outperforms all the baselines and other state-of-the-art methods consistently.
  arXiv  Detail & Related papers  (2024-11-03T02:38:43Z)
• Unified Generative and Discriminative Training for Multi-modal Large Language Models [88.84491005030316]
  Generative training has enabled Vision-Language Models (VLMs) to tackle various complex tasks. Discriminative training, exemplified by models like CLIP, excels in zero-shot image-text classification and retrieval. This paper proposes a unified approach that integrates the strengths of both paradigms.
  arXiv  Detail & Related papers  (2024-11-01T01:51:31Z)
• Effective Tuning Strategies for Generalist Robot Manipulation Policies [45.36380662552082]
  Generalist robot manipulation policies (GMPs) have the potential to generalize across a wide range of tasks, devices, and environments. While fine-tuning offers a practical way to quickly adapt a GMPs to novel domains and tasks with limited samples, we observe that the performance of the resulting GMPs differs significantly with respect to the design choices of fine-tuning strategies.
  arXiv  Detail & Related papers  (2024-10-02T04:00:25Z)
• Towards Generative Class Prompt Learning for Fine-grained Visual Recognition [5.633314115420456]
  Generative Class Prompt Learning and Contrastive Multi-class Prompt Learning are presented. Generative Class Prompt Learning improves visio-linguistic synergy in class embeddings by conditioning on few-shot exemplars with learnable class prompts. CoMPLe builds on this foundation by introducing a contrastive learning component that encourages inter-class separation.
  arXiv  Detail & Related papers  (2024-09-03T12:34:21Z)
• RAR: Retrieving And Ranking Augmented MLLMs for Visual Recognition [78.97487780589574]
  Multimodal Large Language Models (MLLMs) excel at classifying fine-grained categories. This paper introduces a Retrieving And Ranking augmented method for MLLMs. Our proposed approach not only addresses the inherent limitations in fine-grained recognition but also preserves the model's comprehensive knowledge base.
  arXiv  Detail & Related papers  (2024-03-20T17:59:55Z)
• Prompt Perturbation in Retrieval-Augmented Generation based Large Language Models [9.688626139309013]
  Retrieval-Augmented Generation is considered as a means to improve the trustworthiness of text generation from large language models. In this work, we find that the insertion of even a short prefix to the prompt leads to the generation of outputs far away from factually correct answers. We introduce a novel optimization technique called Gradient Guided Prompt Perturbation.
  arXiv  Detail & Related papers  (2024-02-11T12:25:41Z)
• Making LLaMA SEE and Draw with SEED Tokenizer [69.1083058794092]
  We introduce SEED, an elaborate image tokenizer that empowers Large Language Models with the ability to SEE and Draw. With SEED tokens, LLM is able to perform scalable multimodal autoregression under its original training recipe. SEED-LLaMA has exhibited compositional emergent abilities such as multi-turn in-context multimodal generation.
  arXiv  Detail & Related papers  (2023-10-02T14:03:02Z)
• USER: Unified Semantic Enhancement with Momentum Contrast for Image-Text Retrieval [115.28586222748478]
  Image-Text Retrieval (ITR) aims at searching for the target instances that are semantically relevant to the given query from the other modality. Existing approaches typically suffer from two major limitations.
  arXiv  Detail & Related papers  (2023-01-17T12:42:58Z)
• MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognition [118.73025093045652]
  We propose a pre-training model textbfMEmoBERT for multimodal emotion recognition. Unlike the conventional "pre-train, finetune" paradigm, we propose a prompt-based method that reformulates the downstream emotion classification task as a masked text prediction. Our proposed MEmoBERT significantly enhances emotion recognition performance.
  arXiv  Detail & Related papers  (2021-10-27T09:57:00Z)
• Bridging Multi-Task Learning and Meta-Learning: Towards Efficient Training and Effective Adaptation [19.792537914018933]
  Multi-task learning (MTL) aims to improve the generalization of several related tasks by learning them jointly. Modern meta-learning allows unseen tasks with limited labels during the test phase, in the hope of fast adaptation over them. We show that MTL shares the same optimization formulation with a class of gradient-based meta-learning (GBML) algorithms.
  arXiv  Detail & Related papers  (2021-06-16T17:58:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.