MALT: Multi-scale Action Learning Transformer for Online Action Detection

• URL: http://arxiv.org/abs/2405.20892v1
• Date: Fri, 31 May 2024 15:03:35 GMT
• Title: MALT: Multi-scale Action Learning Transformer for Online Action Detection
• Authors: Zhipeng Yang, Ruoyu Wang, Yang Tan, Liping Xie,
• Abstract summary: Online action detection (OAD) aims to identify ongoing actions from streaming video in real-time, without access to future frames. We propose a multi-scale action learning transformer (MALT) that includes a novel recurrent decoder (used for feature fusion) We also introduce an explicit frame scoring mechanism employing sparse attention, which filters irrelevant frames more efficiently.
• Score: 6.819772961626024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online action detection (OAD) aims to identify ongoing actions from streaming video in real-time, without access to future frames. Since these actions manifest at varying scales of granularity, ranging from coarse to fine, projecting an entire set of action frames to a single latent encoding may result in a lack of local information, necessitating the acquisition of action features across multiple scales. In this paper, we propose a multi-scale action learning transformer (MALT), which includes a novel recurrent decoder (used for feature fusion) that includes fewer parameters and can be trained more efficiently. A hierarchical encoder with multiple encoding branches is further proposed to capture multi-scale action features. The output from the preceding branch is then incrementally input to the subsequent branch as part of a cross-attention calculation. In this way, output features transition from coarse to fine as the branches deepen. We also introduce an explicit frame scoring mechanism employing sparse attention, which filters irrelevant frames more efficiently, without requiring an additional network. The proposed method achieved state-of-the-art performance on two benchmark datasets (THUMOS'14 and TVSeries), outperforming all existing models used for comparison, with an mAP of 0.2% for THUMOS'14 and an mcAP of 0.1% for TVseries.

Related papers

• DyFADet: Dynamic Feature Aggregation for Temporal Action Detection [70.37707797523723]
  We build a novel dynamic feature aggregation (DFA) module that can adapt kernel weights and receptive fields at different timestamps. Using DFA helps to develop a Dynamic TAD head (DyHead), which adaptively aggregates the multi-scale features with adjusted parameters. DyFADet, a new dynamic TAD model, achieves promising performance on a series of challenging TAD benchmarks.
  arXiv  Detail & Related papers  (2024-07-03T15:29:10Z)
• Bi-directional Adapter for Multi-modal Tracking [67.01179868400229]
  We propose a novel multi-modal visual prompt tracking model based on a universal bi-directional adapter. We develop a simple but effective light feature adapter to transfer modality-specific information from one modality to another. Our model achieves superior tracking performance in comparison with both the full fine-tuning methods and the prompt learning-based methods.
  arXiv  Detail & Related papers  (2023-12-17T05:27:31Z)
• Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement [83.60486465697318]
  We propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames. We extend an M2M++ framework by introducing a flexible Spatial Selective Refinement component, which allows for trading computational efficiency for quality and vice versa.
  arXiv  Detail & Related papers  (2023-10-29T09:09:32Z)
• BIT: Bi-Level Temporal Modeling for Efficient Supervised Action Segmentation [34.88225099758585]
  supervised action segmentation aims to partition a video into non-overlapping segments, each representing a different action. Recent works apply transformers to perform temporal modeling at the frame-level, which suffer from high computational cost. We propose an efficient BI-level Temporal modeling framework that learns explicit action tokens to represent action segments.
  arXiv  Detail & Related papers  (2023-08-28T20:59:15Z)
• MA-FSAR: Multimodal Adaptation of CLIP for Few-Shot Action Recognition [41.78245303513613]
  We introduce MA-FSAR, a framework that employs the Fine-Tuning (PEFT) technique to enhance the CLIP visual encoder in terms of action-related temporal and semantic representations. In addition to these token-level designs, we propose a prototype-level text-guided construction module to further enrich the temporal and semantic characteristics of video prototypes.
  arXiv  Detail & Related papers  (2023-08-03T04:17:25Z)
• Dynamic Perceiver for Efficient Visual Recognition [87.08210214417309]
  We propose Dynamic Perceiver (Dyn-Perceiver) to decouple the feature extraction procedure and the early classification task. A feature branch serves to extract image features, while a classification branch processes a latent code assigned for classification tasks. Early exits are placed exclusively within the classification branch, thus eliminating the need for linear separability in low-level features.
  arXiv  Detail & Related papers  (2023-06-20T03:00:22Z)
• Look More but Care Less in Video Recognition [57.96505328398205]
  Action recognition methods typically sample a few frames to represent each video to avoid the enormous computation. We propose Ample and Focal Network (AFNet), which is composed of two branches to utilize more frames but with less computation.
  arXiv  Detail & Related papers  (2022-11-18T02:39:56Z)
• Adaptive Perception Transformer for Temporal Action Localization [13.735402329482719]
  This paper proposes a novel end-to-end model, called adaptive perception transformer (AdaPerFormer) One branch takes care of the global perception attention, which can model entire video sequences and aggregate global relevant contexts. The other branch concentrates on the local convolutional shift to aggregate intra-frame and inter-frame information.
  arXiv  Detail & Related papers  (2022-08-25T07:42:48Z)
• Efficient Two-Stream Network for Violence Detection Using Separable Convolutional LSTM [0.0]
  We propose an efficient two-stream deep learning architecture leveraging Separable Convolutional LSTM (SepConvLSTM) and pre-trained MobileNet. SepConvLSTM is constructed by replacing convolution operation at each gate of ConvLSTM with a depthwise separable convolution. Our model outperforms the accuracy on the larger and more challenging RWF-2000 dataset by more than a 2% margin.
  arXiv  Detail & Related papers  (2021-02-21T12:01:48Z)
• Motion-Attentive Transition for Zero-Shot Video Object Segmentation [99.44383412488703]
  We present a Motion-Attentive Transition Network (MATNet) for zero-shot object segmentation. An asymmetric attention block, called Motion-Attentive Transition (MAT), is designed within a two-stream encoder. In this way, the encoder becomes deeply internative, allowing for closely hierarchical interactions between object motion and appearance.
  arXiv  Detail & Related papers  (2020-03-09T16:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.