TIM: A Time Interval Machine for Audio-Visual Action Recognition

• URL: http://arxiv.org/abs/2404.05559v2
• Date: Tue, 9 Apr 2024 07:43:29 GMT
• Title: TIM: A Time Interval Machine for Audio-Visual Action Recognition
• Authors: Jacob Chalk, Jaesung Huh, Evangelos Kazakos, Andrew Zisserman, Dima Damen,
• Abstract summary: We address the interplay between the two modalities in long videos by explicitly modelling the temporal extents of audio and visual events. We propose the Time Interval Machine (TIM) where a modality-specific time interval poses as a query to a transformer encoder. We test TIM on three long audio-visual video datasets: EPIC-KITCHENS, Perception Test, and AVE.
• Score: 64.24297230981168
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diverse actions give rise to rich audio-visual signals in long videos. Recent works showcase that the two modalities of audio and video exhibit different temporal extents of events and distinct labels. We address the interplay between the two modalities in long videos by explicitly modelling the temporal extents of audio and visual events. We propose the Time Interval Machine (TIM) where a modality-specific time interval poses as a query to a transformer encoder that ingests a long video input. The encoder then attends to the specified interval, as well as the surrounding context in both modalities, in order to recognise the ongoing action. We test TIM on three long audio-visual video datasets: EPIC-KITCHENS, Perception Test, and AVE, reporting state-of-the-art (SOTA) for recognition. On EPIC-KITCHENS, we beat previous SOTA that utilises LLMs and significantly larger pre-training by 2.9% top-1 action recognition accuracy. Additionally, we show that TIM can be adapted for action detection, using dense multi-scale interval queries, outperforming SOTA on EPIC-KITCHENS-100 for most metrics, and showing strong performance on the Perception Test. Our ablations show the critical role of integrating the two modalities and modelling their time intervals in achieving this performance. Code and models at: https://github.com/JacobChalk/TIM

Related papers

• OMCAT: Omni Context Aware Transformer [27.674943980306423]
  OCTAV is a novel dataset designed to capture event transitions across audio and video. OMCAT is a powerful model that leverages RoTE to enhance temporal grounding and computational efficiency in time-anchored tasks. Our model demonstrates state-of-the-art performance on Audio-Visual Question Answering (AVQA) tasks and the OCTAV benchmark, showcasing significant gains in temporal reasoning and cross-modal alignment.
  arXiv  Detail & Related papers  (2024-10-15T23:16:28Z)
• Harnessing Temporal Causality for Advanced Temporal Action Detection [53.654457142657236]
  We introduce CausalTAD, which combines causal attention and causal Mamba to achieve state-of-the-art performance on benchmarks. We ranked 1st in the Action Recognition, Action Detection, and Audio-Based Interaction Detection tracks at the EPIC-Kitchens Challenge 2024, and 1st in the Moment Queries track at the Ego4D Challenge 2024.
  arXiv  Detail & Related papers  (2024-07-25T06:03:02Z)
• EchoTrack: Auditory Referring Multi-Object Tracking for Autonomous Driving [64.58258341591929]
  Auditory Referring Multi-Object Tracking (AR-MOT) is a challenging problem in autonomous driving. We put forward EchoTrack, an end-to-end AR-MOT framework with dual-stream vision transformers. We establish the first set of large-scale AR-MOT benchmarks.
  arXiv  Detail & Related papers  (2024-02-28T12:50:16Z)
• Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities [67.89368528234394]
  One of the main challenges of multimodal learning is the need to combine heterogeneous modalities. Video and audio are obtained at much higher rates than text and are roughly aligned in time. Our approach achieves the state-of-the-art on well established multimodal benchmarks, outperforming much larger models.
  arXiv  Detail & Related papers  (2023-11-09T19:15:12Z)
• Audio-Visual Instance Segmentation [14.10809424760213]
  We propose a new multi-modal task, termed audio-visual instance segmentation (AVIS) AVIS aims to simultaneously identify, segment and track individual sounding object instances in audible videos. We introduce a high-quality benchmark named AVISeg, containing over 90K instance masks from 26 semantic categories in 926 long videos.
  arXiv  Detail & Related papers  (2023-10-28T13:37:52Z)
• TMac: Temporal Multi-Modal Graph Learning for Acoustic Event Classification [60.038979555455775]
  We propose a Temporal Multi-modal graph learning method for Acoustic event Classification, called TMac. In particular, we construct a temporal graph for each acoustic event, dividing its audio data and video data into multiple segments. Several experiments are conducted to demonstrate TMac outperforms other SOTA models in performance.
  arXiv  Detail & Related papers  (2023-09-21T07:39:08Z)
• End-To-End Audiovisual Feature Fusion for Active Speaker Detection [7.631698269792165]
  This work presents a novel two-stream end-to-end framework fusing features extracted from images via VGG-M with raw Mel Frequency Cepstrum Coefficients features extracted from the audio waveform. Our best-performing model attained 88.929% accuracy, nearly the same detection result as state-of-the-art -work.
  arXiv  Detail & Related papers  (2022-07-27T10:25:59Z)
• Slow-Fast Visual Tempo Learning for Video-based Action Recognition [78.3820439082979]
  Action visual tempo characterizes the dynamics and the temporal scale of an action. Previous methods capture the visual tempo either by sampling raw videos with multiple rates, or by hierarchically sampling backbone features. We propose a Temporal Correlation Module (TCM) to extract action visual tempo from low-level backbone features at single-layer remarkably.
  arXiv  Detail & Related papers  (2022-02-24T14:20:04Z)
• Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoders [29.658535633701035]
  We propose a novel conditional variational autoencoder (VAE) that explicitly models one-to-many audio-to-motion mapping. We show that our method generates more realistic and diverse motions than state-of-the-art methods, quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2021-08-15T11:15:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.