Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes

• URL: http://arxiv.org/abs/2311.17948v2
• Date: Sat, 20 Apr 2024 15:29:00 GMT
• Title: Action-slot: Visual Action-centric Representations for Multi-label Atomic Activity Recognition in Traffic Scenes
• Authors: Chi-Hsi Kung, Shu-Wei Lu, Yi-Hsuan Tsai, Yi-Ting Chen,
• Abstract summary: Action-slot is a slot attention-based approach that learns visual action-centric representations. Our key idea is to design action slots that are capable of paying attention to regions where atomic activities occur. To address the limitation, we collect a synthetic dataset called TACO, which is four times larger than OATS.
• Score: 23.284478293459856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we study multi-label atomic activity recognition. Despite the notable progress in action recognition, it is still challenging to recognize atomic activities due to a deficiency in a holistic understanding of both multiple road users' motions and their contextual information. In this paper, we introduce Action-slot, a slot attention-based approach that learns visual action-centric representations, capturing both motion and contextual information. Our key idea is to design action slots that are capable of paying attention to regions where atomic activities occur, without the need for explicit perception guidance. To further enhance slot attention, we introduce a background slot that competes with action slots, aiding the training process in avoiding unnecessary focus on background regions devoid of activities. Yet, the imbalanced class distribution in the existing dataset hampers the assessment of rare activities. To address the limitation, we collect a synthetic dataset called TACO, which is four times larger than OATS and features a balanced distribution of atomic activities. To validate the effectiveness of our method, we conduct comprehensive experiments and ablation studies against various action recognition baselines. We also show that the performance of multi-label atomic activity recognition on real-world datasets can be improved by pretraining representations on TACO. We will release our source code and dataset. See the videos of visualization on the project page: https://hcis-lab.github.io/Action-slot/

Related papers

• ARIC: An Activity Recognition Dataset in Classroom Surveillance Images [19.586321497367294]
  We construct a novel dataset focused on classroom surveillance image activity recognition called ARIC (Activity Recognition In Classroom) The ARIC dataset has advantages of multiple perspectives, 32 activity categories, three modalities, and real-world classroom scenarios. We hope that the ARIC dataset can act as a facilitator for future analysis and research for open teaching scenarios.
  arXiv  Detail & Related papers  (2024-10-16T07:59:07Z)
• Few-Shot Continual Learning for Activity Recognition in Classroom Surveillance Images [13.328067147864092]
  In real classroom settings, normal teaching activities account for a large proportion of samples, while rare non-teaching activities such as eating continue to appear. This requires a model that can learn non-teaching activities from few samples without forgetting the normal teaching activities.
  arXiv  Detail & Related papers  (2024-09-05T08:55:56Z)
• VCHAR:Variance-Driven Complex Human Activity Recognition framework with Generative Representation [6.278293754210117]
  VCHAR (Variance-Driven Complex Human Activity Recognition) is a novel framework that treats the outputs of atomic activities as a distribution over specified intervals. We show that VCHAR enhances the accuracy of complex activity recognition without necessitating precise temporal or sequential labeling of atomic activities.
  arXiv  Detail & Related papers  (2024-07-03T17:24:36Z)
• Object-centric Cross-modal Feature Distillation for Event-based Object Detection [87.50272918262361]
  RGB detectors still outperform event-based detectors due to sparsity of the event data and missing visual details. We develop a novel knowledge distillation approach to shrink the performance gap between these two modalities. We show that object-centric distillation allows to significantly improve the performance of the event-based student object detector.
  arXiv  Detail & Related papers  (2023-11-09T16:33:08Z)
• TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learning [73.53576440536682]
  We introduce TACO: Temporal Action-driven Contrastive Learning, a powerful temporal contrastive learning approach. TACO simultaneously learns a state and an action representation by optimizing the mutual information between representations of current states. For online RL, TACO achieves 40% performance boost after one million environment interaction steps.
  arXiv  Detail & Related papers  (2023-06-22T22:21:53Z)
• Few-Shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning [51.03781020616402]
  Fine-grained action recognition is attracting increasing attention due to the emerging demand of specific action understanding in real-world applications. We propose a few-shot fine-grained action recognition problem, aiming to recognize novel fine-grained actions with only few samples given for each class. Although progress has been made in coarse-grained actions, existing few-shot recognition methods encounter two issues handling fine-grained actions.
  arXiv  Detail & Related papers  (2021-08-15T02:21:01Z)
• Home Action Genome: Cooperative Compositional Action Understanding [33.69990813932372]
  Existing research on action recognition treats activities as monolithic events occurring in videos. Cooperative Compositional Action Understanding (CCAU) is a cooperative learning framework for hierarchical action recognition. We demonstrate the utility of co-learning compositions in few-shot action recognition by achieving 28.6% mAP with just a single sample.
  arXiv  Detail & Related papers  (2021-05-11T17:42:47Z)
• Semi-Supervised Few-Shot Atomic Action Recognition [59.587738451616495]
  We propose a novel model for semi-supervised few-shot atomic action recognition. Our model features unsupervised and contrastive video embedding, loose action alignment, multi-head feature comparison, and attention-based aggregation. Experiments show that our model can attain high accuracy on representative atomic action datasets outperforming their respective state-of-the-art classification accuracy in full supervision setting.
  arXiv  Detail & Related papers  (2020-11-17T03:59:05Z)
• Intra- and Inter-Action Understanding via Temporal Action Parsing [118.32912239230272]
  We construct a new dataset developed on sport videos with manual annotations of sub-actions, and conduct a study on temporal action parsing on top. Our study shows that a sport activity usually consists of multiple sub-actions and that the awareness of such temporal structures is beneficial to action recognition. We also investigate a number of temporal parsing methods, and thereon devise an improved method that is capable of mining sub-actions from training data without knowing the labels of them.
  arXiv  Detail & Related papers  (2020-05-20T17:45:18Z)
• ZSTAD: Zero-Shot Temporal Activity Detection [107.63759089583382]
  We propose a novel task setting called zero-shot temporal activity detection (ZSTAD), where activities that have never been seen in training can still be detected. We design an end-to-end deep network based on R-C3D as the architecture for this solution. Experiments on both the THUMOS14 and the Charades datasets show promising performance in terms of detecting unseen activities.
  arXiv  Detail & Related papers  (2020-03-12T02:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.