Score-balanced Loss for Multi-aspect Pronunciation Assessment

• URL: http://arxiv.org/abs/2305.16664v1
• Date: Fri, 26 May 2023 06:21:37 GMT
• Title: Score-balanced Loss for Multi-aspect Pronunciation Assessment
• Authors: Heejin Do, Yunsu Kim, Gary Geunbae Lee
• Abstract summary: We propose a novel loss function, score-balanced loss, to address the problem caused by uneven data. As a re-weighting approach, we assign higher costs when the predicted score is of the minority class. We evaluate our method on the speechocean762 dataset, which has noticeably imbalanced scores for several aspects.
• Score: 3.6825890616838066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With rapid technological growth, automatic pronunciation assessment has transitioned toward systems that evaluate pronunciation in various aspects, such as fluency and stress. However, despite the highly imbalanced score labels within each aspect, existing studies have rarely tackled the data imbalance problem. In this paper, we suggest a novel loss function, score-balanced loss, to address the problem caused by uneven data, such as bias toward the majority scores. As a re-weighting approach, we assign higher costs when the predicted score is of the minority class, thus, guiding the model to gain positive feedback for sparse score prediction. Specifically, we design two weighting factors by leveraging the concept of an effective number of samples and using the ranks of scores. We evaluate our method on the speechocean762 dataset, which has noticeably imbalanced scores for several aspects. Improved results particularly on such uneven aspects prove the effectiveness of our method.

Related papers

• Revising the Problem of Partial Labels from the Perspective of CNNs' Robustness [6.46250754192468]
  We introduce a lightweight partial-label solution using pseudo-labeling techniques and a designed loss function. We employ D-Score to analyze both the proposed and existing methods to determine whether they can enhance robustness while improving accuracy.
  arXiv  Detail & Related papers  (2024-07-24T20:39:17Z)
• Fighting Sampling Bias: A Framework for Training and Evaluating Credit Scoring Models [2.918530881730374]
  This paper addresses the adverse effect of sampling bias on model training and evaluation. We propose bias-aware self-learning and a reject inference framework for scorecard evaluation. Our results suggest a profit improvement of about eight percent, when using Bayesian evaluation to decide on acceptance rates.
  arXiv  Detail & Related papers  (2024-07-17T20:59:54Z)
• Acoustic Feature Mixup for Balanced Multi-aspect Pronunciation Assessment [7.519788903817844]
  We propose two Acoustic Feature Mixup strategies to address data scarcity and score-label imbalances. We integrate fine-grained error-rate features by comparing speech recognition results with the original answer phonemes, giving direct hints for mispronunciation.
  arXiv  Detail & Related papers  (2024-06-22T03:56:29Z)
• Learning with Imbalanced Noisy Data by Preventing Bias in Sample Selection [82.43311784594384]
  Real-world datasets contain not only noisy labels but also class imbalance. We propose a simple yet effective method to address noisy labels in imbalanced datasets.
  arXiv  Detail & Related papers  (2024-02-17T10:34:53Z)
• Beyond Myopia: Learning from Positive and Unlabeled Data through Holistic Predictive Trends [26.79150786180822]
  We unveil an intriguing yet long-overlooked observation in PUL. Predictive trends for positive and negative classes display distinctly different patterns. We propose a novel TPP-inspired measure for trend detection and prove its unbiasedness in predicting changes.
  arXiv  Detail & Related papers  (2023-10-06T08:06:15Z)
• Uncertainty-Driven Action Quality Assessment [67.20617610820857]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores. We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss. Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Abnormal-aware Multi-person Evaluation System with Improved Fuzzy Weighting [0.0]
  We choose the two-stage screening method, which consists of rough screening and score-weighted Kendall-$tau$ Distance. We use Fuzzy Synthetic Evaluation Method(FSE) to determine the significance of scores given by reviewers as well as their reliability.
  arXiv  Detail & Related papers  (2022-05-01T03:42:43Z)
• Scale-Equivalent Distillation for Semi-Supervised Object Detection [57.59525453301374]
  Recent Semi-Supervised Object Detection (SS-OD) methods are mainly based on self-training, generating hard pseudo-labels by a teacher model on unlabeled data as supervisory signals. We analyze the challenges these methods meet with the empirical experiment results. We introduce a novel approach, Scale-Equivalent Distillation (SED), which is a simple yet effective end-to-end knowledge distillation framework robust to large object size variance and class imbalance.
  arXiv  Detail & Related papers  (2022-03-23T07:33:37Z)
• Disentangling Sampling and Labeling Bias for Learning in Large-Output Spaces [64.23172847182109]
  We show that different negative sampling schemes implicitly trade-off performance on dominant versus rare labels. We provide a unified means to explicitly tackle both sampling bias, arising from working with a subset of all labels, and labeling bias, which is inherent to the data due to label imbalance.
  arXiv  Detail & Related papers  (2021-05-12T15:40:13Z)
• Counterfactual Representation Learning with Balancing Weights [74.67296491574318]
  Key to causal inference with observational data is achieving balance in predictive features associated with each treatment type. Recent literature has explored representation learning to achieve this goal. We develop an algorithm for flexible, scalable and accurate estimation of causal effects.
  arXiv  Detail & Related papers  (2020-10-23T19:06:03Z)
• Uncertainty-aware Score Distribution Learning for Action Quality Assessment [91.05846506274881]
  We propose an uncertainty-aware score distribution learning (USDL) approach for action quality assessment (AQA) Specifically, we regard an action as an instance associated with a score distribution, which describes the probability of different evaluated scores. Under the circumstance where fine-grained score labels are available, we devise a multi-path uncertainty-aware score distributions learning (MUSDL) method to explore the disentangled components of a score.
  arXiv  Detail & Related papers  (2020-06-13T15:41:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.