ContextDet: Temporal Action Detection with Adaptive Context Aggregation

• URL: http://arxiv.org/abs/2410.15279v1
• Date: Sun, 20 Oct 2024 04:28:19 GMT
• Title: ContextDet: Temporal Action Detection with Adaptive Context Aggregation
• Authors: Ning Wang, Yun Xiao, Xiaopeng Peng, Xiaojun Chang, Xuanhong Wang, Dingyi Fang,
• Abstract summary: We introduce a single-stage ContextDet framework for temporal action detection (TAD) Our model features a pyramid adaptive context aggragation (ACA) architecture, capturing long context and improving action discriminability. By varying the length of these large kernels across the ACA pyramid, our model provides lightweight yet effective context aggregation and action discrimination.
• Score: 47.84334557998388
• License:
• Abstract: Temporal action detection (TAD), which locates and recognizes action segments, remains a challenging task in video understanding due to variable segment lengths and ambiguous boundaries. Existing methods treat neighboring contexts of an action segment indiscriminately, leading to imprecise boundary predictions. We introduce a single-stage ContextDet framework, which makes use of large-kernel convolutions in TAD for the first time. Our model features a pyramid adaptive context aggragation (ACA) architecture, capturing long context and improving action discriminability. Each ACA level consists of two novel modules. The context attention module (CAM) identifies salient contextual information, encourages context diversity, and preserves context integrity through a context gating block (CGB). The long context module (LCM) makes use of a mixture of large- and small-kernel convolutions to adaptively gather long-range context and fine-grained local features. Additionally, by varying the length of these large kernels across the ACA pyramid, our model provides lightweight yet effective context aggregation and action discrimination. We conducted extensive experiments and compared our model with a number of advanced TAD methods on six challenging TAD benchmarks: MultiThumos, Charades, FineAction, EPIC-Kitchens 100, Thumos14, and HACS, demonstrating superior accuracy at reduced inference speed.

Related papers

• Contextrast: Contextual Contrastive Learning for Semantic Segmentation [9.051352746190448]
  We propose Contextrast, a contrastive learning-based semantic segmentation method. Our proposed method comprises two parts: a) contextual contrastive learning (CCL) and b) boundary-aware negative sampling. We demonstrate that our Contextrast substantially enhances the performance of semantic segmentation networks.
  arXiv  Detail & Related papers  (2024-04-16T15:04:55Z)
• ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detection [10.012716326383567]
  Temporal action detection (TAD) involves the localization and classification of action instances within untrimmed videos. We present ZEETAD, featuring two modules: dual-localization and zero-shot proposal classification. We enhance discriminative capability on unseen classes by minimally updating the frozen CLIP encoder with lightweight adapters.
  arXiv  Detail & Related papers  (2023-11-01T00:17:37Z)
• Temporal Action Localization with Enhanced Instant Discriminability [66.76095239972094]
  Temporal action detection (TAD) aims to detect all action boundaries and their corresponding categories in an untrimmed video. We propose a one-stage framework named TriDet to resolve imprecise predictions of action boundaries by existing methods. Experimental results demonstrate the robustness of TriDet and its state-of-the-art performance on multiple TAD datasets.
  arXiv  Detail & Related papers  (2023-09-11T16:17:50Z)
• Context-aware Fine-tuning of Self-supervised Speech Models [56.95389222319555]
  We study the use of context, i.e., surrounding segments, during fine-tuning. We propose a new approach called context-aware fine-tuning. We evaluate the proposed approach using the SLUE and Libri-light benchmarks for several downstream tasks.
  arXiv  Detail & Related papers  (2022-12-16T15:46:15Z)
• Learning Commonsense-aware Moment-Text Alignment for Fast Video Temporal Grounding [78.71529237748018]
  Grounding temporal video segments described in natural language queries effectively and efficiently is a crucial capability needed in vision-and-language fields. Most existing approaches adopt elaborately designed cross-modal interaction modules to improve the grounding performance. We propose a commonsense-aware cross-modal alignment framework, which incorporates commonsense-guided visual and text representations into a complementary common space.
  arXiv  Detail & Related papers  (2022-04-04T13:07:05Z)
• Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization [87.47977407022492]
  This paper argues that learning by contextually comparing sequence-to-sequence distinctions offers an essential inductive bias in weakly-supervised action localization. Under a differentiable dynamic programming formulation, two complementary contrastive objectives are designed, including Fine-grained Sequence Distance (FSD) contrasting and Longest Common Subsequence (LCS) contrasting. Our method achieves state-of-the-art performance on two popular benchmarks.
  arXiv  Detail & Related papers  (2022-03-31T05:13:50Z)
• Hierarchical Deep Residual Reasoning for Temporal Moment Localization [48.108468456043994]
  We propose a Hierarchical Deep Residual Reasoning (HDRR) model, which decomposes the video and sentence into multi-level representations with different semantics. We also design the simple yet effective Res-BiGRUs for feature fusion, which is able to grasp the useful information in a self-adapting manner.
  arXiv  Detail & Related papers  (2021-10-31T07:13:34Z)
• A Unified Efficient Pyramid Transformer for Semantic Segmentation [40.20512714144266]
  We advocate a unified framework(UN-EPT) to segment objects by considering both context information and boundary artifacts. We first adapt a sparse sampling strategy to incorporate the transformer-based attention mechanism for efficient context modeling. We demonstrate promising performance on three popular benchmarks for semantic segmentation with low memory footprint.
  arXiv  Detail & Related papers  (2021-07-29T17:47:32Z)
• Learning Salient Boundary Feature for Anchor-free Temporal Action Localization [81.55295042558409]
  Temporal action localization is an important yet challenging task in video understanding. We propose the first purely anchor-free temporal localization method. Our model includes (i) an end-to-end trainable basic predictor, (ii) a saliency-based refinement module, and (iii) several consistency constraints.
  arXiv  Detail & Related papers  (2021-03-24T12:28:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.