DyFADet: Dynamic Feature Aggregation for Temporal Action Detection

• URL: http://arxiv.org/abs/2407.03197v1
• Date: Wed, 3 Jul 2024 15:29:10 GMT
• Title: DyFADet: Dynamic Feature Aggregation for Temporal Action Detection
• Authors: Le Yang, Ziwei Zheng, Yizeng Han, Hao Cheng, Shiji Song, Gao Huang, Fan Li,
• Abstract summary: We build a novel dynamic feature aggregation (DFA) module that can adapt kernel weights and receptive fields at different timestamps. Using DFA helps to develop a Dynamic TAD head (DyHead), which adaptively aggregates the multi-scale features with adjusted parameters. DyFADet, a new dynamic TAD model, achieves promising performance on a series of challenging TAD benchmarks.
• Score: 70.37707797523723
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent proposed neural network-based Temporal Action Detection (TAD) models are inherently limited to extracting the discriminative representations and modeling action instances with various lengths from complex scenes by shared-weights detection heads. Inspired by the successes in dynamic neural networks, in this paper, we build a novel dynamic feature aggregation (DFA) module that can simultaneously adapt kernel weights and receptive fields at different timestamps. Based on DFA, the proposed dynamic encoder layer aggregates the temporal features within the action time ranges and guarantees the discriminability of the extracted representations. Moreover, using DFA helps to develop a Dynamic TAD head (DyHead), which adaptively aggregates the multi-scale features with adjusted parameters and learned receptive fields better to detect the action instances with diverse ranges from videos. With the proposed encoder layer and DyHead, a new dynamic TAD model, DyFADet, achieves promising performance on a series of challenging TAD benchmarks, including HACS-Segment, THUMOS14, ActivityNet-1.3, Epic-Kitchen 100, Ego4D-Moment QueriesV1.0, and FineAction. Code is released to https://github.com/yangle15/DyFADet-pytorch.

Related papers

• Slow-Fast Visual Tempo Learning for Video-based Action Recognition [78.3820439082979]
  Action visual tempo characterizes the dynamics and the temporal scale of an action. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, or by hierarchically sampling backbone features. We propose a Temporal Correlation Module (TCM) to extract action visual tempo from low-level backbone features at single-layer remarkably.
  arXiv  Detail & Related papers  (2022-02-24T14:20:04Z)
• Variational Predictive Routing with Nested Subjective Timescales [1.6114012813668934]
  We present Variational Predictive Routing (PRV) - a neural inference system that organizes latent video features in a temporal hierarchy. We show that VPR is able to detect event boundaries, disentangletemporal features, adapt to the dynamics hierarchy of the data, and produce accurate time-agnostic rollouts of the future.
  arXiv  Detail & Related papers  (2021-10-21T16:12:59Z)
• MFGNet: Dynamic Modality-Aware Filter Generation for RGB-T Tracking [72.65494220685525]
  We propose a new dynamic modality-aware filter generation module (named MFGNet) to boost the message communication between visible and thermal data. We generate dynamic modality-aware filters with two independent networks. The visible and thermal filters will be used to conduct a dynamic convolutional operation on their corresponding input feature maps respectively. To address issues caused by heavy occlusion, fast motion, and out-of-view, we propose to conduct a joint local and global search by exploiting a new direction-aware target-driven attention mechanism.
  arXiv  Detail & Related papers  (2021-07-22T03:10:51Z)
• TDN: Temporal Difference Networks for Efficient Action Recognition [31.922001043405924]
  This paper presents a new video architecture, termed as Temporal Difference Network (TDN) The core of our TDN is to devise an efficient temporal module (TDM) by explicitly leveraging a temporal difference operator. Our TDN presents a new state of the art on the Something-Something V1 & V2 datasets and is on par with the best performance on the Kinetics-400 dataset.
  arXiv  Detail & Related papers  (2020-12-18T06:31:08Z)
• Fine-Grained Dynamic Head for Object Detection [68.70628757217939]
  We propose a fine-grained dynamic head to conditionally select a pixel-level combination of FPN features from different scales for each instance. Experiments demonstrate the effectiveness and efficiency of the proposed method on several state-of-the-art detection benchmarks.
  arXiv  Detail & Related papers  (2020-12-07T08:16:32Z)
• Learn to cycle: Time-consistent feature discovery for action recognition [83.43682368129072]
  Generalizing over temporal variations is a prerequisite for effective action recognition in videos. We introduce Squeeze Re Temporal Gates (SRTG), an approach that favors temporal activations with potential variations. We show consistent improvement when using SRTPG blocks, with only a minimal increase in the number of GFLOs.
  arXiv  Detail & Related papers  (2020-06-15T09:36:28Z)
• TAM: Temporal Adaptive Module for Video Recognition [60.83208364110288]
  temporal adaptive module (bf TAM) generates video-specific temporal kernels based on its own feature map. Experiments on Kinetics-400 and Something-Something datasets demonstrate that our TAM outperforms other temporal modeling methods consistently.
  arXiv  Detail & Related papers  (2020-05-14T08:22:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.