Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation

• URL: http://arxiv.org/abs/2503.18358v1
• Date: Mon, 24 Mar 2025 05:37:12 GMT
• Title: Cost-Sensitive Learning for Long-Tailed Temporal Action Segmentation
• Authors: Zhanzhong Pang, Fadime Sener, Shrinivas Ramasubramanian, Angela Yao,
• Abstract summary: Temporal action segmentation in procedural videos aims to densely label frames into action classes.<n>We identify a bi-level learning bias, stemming from class imbalance favoring head classes, and a transition-level bias arising from variations in transitions.<n>As a remedy, we introduce a constrained optimization problem to alleviate both biases.
• Score: 35.710284726160815
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Temporal action segmentation in untrimmed procedural videos aims to densely label frames into action classes. These videos inherently exhibit long-tailed distributions, where actions vary widely in frequency and duration. In temporal action segmentation approaches, we identified a bi-level learning bias. This bias encompasses (1) a class-level bias, stemming from class imbalance favoring head classes, and (2) a transition-level bias arising from variations in transitions, prioritizing commonly observed transitions. As a remedy, we introduce a constrained optimization problem to alleviate both biases. We define learning states for action classes and their associated transitions and integrate them into the optimization process. We propose a novel cost-sensitive loss function formulated as a weighted cross-entropy loss, with weights adaptively adjusted based on the learning state of actions and their transitions. Experiments on three challenging temporal segmentation benchmarks and various frameworks demonstrate the effectiveness of our approach, resulting in significant improvements in both per-class frame-wise and segment-wise performance.

Related papers

• Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment [33.74853437611066]
  Weakly-supervised action segmentation is a task of learning to partition a long video into several action segments, where training videos are only accompanied by transcripts. Most of existing methods need to infer pseudo segmentation for training by serial alignment between all frames and the transcript. We propose a novel Action-Transition-Aware Boundary Alignment framework to efficiently and effectively filter out noisy boundaries and detect transitions.
  arXiv  Detail & Related papers  (2024-03-28T08:39:44Z)
• Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization [87.47977407022492]
  This paper argues that learning by contextually comparing sequence-to-sequence distinctions offers an essential inductive bias in weakly-supervised action localization. Under a differentiable dynamic programming formulation, two complementary contrastive objectives are designed, including Fine-grained Sequence Distance (FSD) contrasting and Longest Common Subsequence (LCS) contrasting. Our method achieves state-of-the-art performance on two popular benchmarks.
  arXiv  Detail & Related papers  (2022-03-31T05:13:50Z)
• Temporal Transductive Inference for Few-Shot Video Object Segmentation [27.140141181513425]
  Few-shot object segmentation (FS-VOS) aims at segmenting video frames using a few labelled examples of classes not seen during initial training. Key to our approach is the use of both global and local temporal constraints. Empirically, our model outperforms state-of-the-art meta-learning approaches in terms of mean intersection over union on YouTube-VIS by 2.8%.
  arXiv  Detail & Related papers  (2022-03-27T14:08:30Z)
• Efficient Modelling Across Time of Human Actions and Interactions [92.39082696657874]
  We argue that current fixed-sized-temporal kernels in 3 convolutional neural networks (CNNDs) can be improved to better deal with temporal variations in the input. We study how we can better handle between classes of actions, by enhancing their feature differences over different layers of the architecture. The proposed approaches are evaluated on several benchmark action recognition datasets and show competitive results.
  arXiv  Detail & Related papers  (2021-10-05T15:39:11Z)
• Weakly Supervised Temporal Action Localization with Segment-Level Labels [140.68096218667162]
  Temporal action localization presents a trade-off between test performance and annotation-time cost. We introduce a new segment-level supervision setting: segments are labeled when annotators observe actions happening here. We devise a partial segment loss regarded as a loss sampling to learn integral action parts from labeled segments.
  arXiv  Detail & Related papers  (2020-07-03T10:32:19Z)
• On Evaluating Weakly Supervised Action Segmentation Methods [79.42955857919497]
  We focus on two aspects of the use and evaluation of weakly supervised action segmentation approaches. We train each method on the Breakfast dataset 5 times and provide average and standard deviation of the results. Our experiments show that the standard deviation over these repetitions is between 1 and 2.5% and significantly affects the comparison between different approaches.
  arXiv  Detail & Related papers  (2020-05-19T20:30:31Z)
• Bottom-Up Temporal Action Localization with Mutual Regularization [107.39785866001868]
  State-of-the-art solutions for TAL involve evaluating the frame-level probabilities of three action-indicating phases. We introduce two regularization terms to mutually regularize the learning procedure. Experiments are performed on two popular TAL datasets, THUMOS14 and ActivityNet1.3.
  arXiv  Detail & Related papers  (2020-02-18T03:59:13Z)
• Weakly Supervised Temporal Action Localization Using Deep Metric Learning [12.49814373580862]
  We propose a weakly supervised temporal action localization method that only requires video-level action instances as supervision during training. We jointly optimize a balanced binary cross-entropy loss and a metric loss using a standard backpropagation algorithm. Our approach improves the current state-of-the-art result for THUMOS14 by 6.5% mAP at IoU threshold 0.5, and achieves competitive performance for ActivityNet1.2.
  arXiv  Detail & Related papers  (2020-01-21T22:01:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.