DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding

• URL: http://arxiv.org/abs/2506.23196v1
• Date: Sun, 29 Jun 2025 11:50:19 GMT
• Title: DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding
• Authors: Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert,
• Abstract summary: We introduce DEL, a framework for dense semantic action localization.<n> DEL aims to accurately detect and classify multiple actions at fine-grained temporal resolutions in long untrimmed videos.
• Score: 13.256830504062332
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world videos often contain overlapping events and complex temporal dependencies, making multimodal interaction modeling particularly challenging. We introduce DEL, a framework for dense semantic action localization, aiming to accurately detect and classify multiple actions at fine-grained temporal resolutions in long untrimmed videos. DEL consists of two key modules: the alignment of audio and visual features that leverage masked self-attention to enhance intra-mode consistency and a multimodal interaction refinement module that models cross-modal dependencies across multiple scales, enabling high-level semantics and fine-grained details. Our method achieves state-of-the-art performance on multiple real-world Temporal Action Localization (TAL) datasets, UnAV-100, THUMOS14, ActivityNet 1.3, and EPIC-Kitchens-100, surpassing previous approaches with notable average mAP gains of +3.3%, +2.6%, +1.2%, +1.7% (verb), and +1.4% (noun), respectively.

Related papers

• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• TEM^3-Learning: Time-Efficient Multimodal Multi-Task Learning for Advanced Assistive Driving [22.22943635900334]
  TEM3-Learning is a novel framework that jointly optimize driver emotion recognition, driver behavior recognition, traffic context recognition, and vehicle behavior recognition.<n>It achieves state-of-the-art accuracy across all four tasks, maintaining a lightweight architecture with fewer than 6 million parameters and delivering an impressive 142.32 FPS inference speed.
  arXiv  Detail & Related papers  (2025-06-22T16:12:27Z)
• SM3Det: A Unified Model for Multi-Modal Remote Sensing Object Detection [73.49799596304418]
  This paper introduces a new task called Multi-Modal datasets and Multi-Task Object Detection (M2Det) for remote sensing.<n>It is designed to accurately detect horizontal or oriented objects from any sensor modality.<n>This task poses challenges due to 1) the trade-offs involved in managing multi-modal modelling and 2) the complexities of multi-task optimization.
  arXiv  Detail & Related papers  (2024-12-30T02:47:51Z)
• Learning Modality-agnostic Representation for Semantic Segmentation from Any Modalities [8.517830626176641]
  Any2Seg is a novel framework that can achieve robust segmentation from any combination of modalities in any visual conditions. Experiments on two benchmarks with four modalities demonstrate that Any2Seg achieves the state-of-the-art under the multi-modal setting.
  arXiv  Detail & Related papers  (2024-07-16T03:34:38Z)
• U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation [63.31007867379312]
  We introduce U3M: An Unbiased Multiscale Modal Fusion Model for Multimodal Semantics. We employ feature fusion at multiple scales to ensure the effective extraction and integration of both global and local features. Experimental results demonstrate that our approach achieves superior performance across multiple datasets.
  arXiv  Detail & Related papers  (2024-05-24T08:58:48Z)
• Joint Multimodal Transformer for Emotion Recognition in the Wild [49.735299182004404]
  Multimodal emotion recognition (MMER) systems typically outperform unimodal systems. This paper proposes an MMER method that relies on a joint multimodal transformer (JMT) for fusion with key-based cross-attention.
  arXiv  Detail & Related papers  (2024-03-15T17:23:38Z)
• Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding [62.70450216120704]
  Unsupervised pre-training has shown great success in skeleton-based action understanding. We propose a Unified Multimodal Unsupervised Representation Learning framework, called UmURL. UmURL exploits an efficient early-fusion strategy to jointly encode the multi-modal features in a single-stream manner.
  arXiv  Detail & Related papers  (2023-11-06T13:56:57Z)
• AVE-CLIP: AudioCLIP-based Multi-window Temporal Transformer for Audio Visual Event Localization [14.103742565510387]
  We introduce AVE-CLIP, a novel framework that integrates the AudioCLIP pre-trained on large-scale audio-visual data with a multi-window temporal transformer. Our method achieves state-of-the-art performance on the publicly available AVE dataset with 5.9% mean accuracy improvement.
  arXiv  Detail & Related papers  (2022-10-11T00:15:45Z)
• Slow-Fast Visual Tempo Learning for Video-based Action Recognition [78.3820439082979]
  Action visual tempo characterizes the dynamics and the temporal scale of an action. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, or by hierarchically sampling backbone features. We propose a Temporal Correlation Module (TCM) to extract action visual tempo from low-level backbone features at single-layer remarkably.
  arXiv  Detail & Related papers  (2022-02-24T14:20:04Z)
• MS-TCT: Multi-Scale Temporal ConvTransformer for Action Detection [37.25262046781015]
  Action detection is an essential and challenging task, especially for densely labelled datasets of untrimmed videos. We propose a novel ConvTransformer network for action detection that efficiently captures both short-term and long-term temporal information. Our network outperforms the state-of-the-art methods on all three datasets.
  arXiv  Detail & Related papers  (2021-12-07T18:57:37Z)
• GTA: Global Temporal Attention for Video Action Understanding [51.476605514802806]
  We introduce Global Temporal Attention (AGT), which performs global temporal attention on top of spatial attention in a decoupled manner. Tests on 2D and 3D networks demonstrate that our approach consistently enhances temporal modeling and provides state-of-the-art performance on three video action recognition datasets.
  arXiv  Detail & Related papers  (2020-12-15T18:58:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.