Streaming Detection of Queried Event Start

• URL: http://arxiv.org/abs/2412.03567v1
• Date: Wed, 04 Dec 2024 18:58:27 GMT
• Title: Streaming Detection of Queried Event Start
• Authors: Cristobal Eyzaguirre, Eric Tang, Shyamal Buch, Adrien Gaidon, Jiajun Wu, Juan Carlos Niebles,
• Abstract summary: We propose a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES) The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events.
• Score: 41.500121314019864
• License:
• Abstract: Robotics, autonomous driving, augmented reality, and many embodied computer vision applications must quickly react to user-defined events unfolding in real time. We address this setting by proposing a novel task for multimodal video understanding-Streaming Detection of Queried Event Start (SDQES). The goal of SDQES is to identify the beginning of a complex event as described by a natural language query, with high accuracy and low latency. We introduce a new benchmark based on the Ego4D dataset, as well as new task-specific metrics to study streaming multimodal detection of diverse events in an egocentric video setting. Inspired by parameter-efficient fine-tuning methods in NLP and for video tasks, we propose adapter-based baselines that enable image-to-video transfer learning, allowing for efficient online video modeling. We evaluate three vision-language backbones and three adapter architectures on both short-clip and untrimmed video settings.

Related papers

• Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge [57.01131456894516]
  Current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. We propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications.
  arXiv  Detail & Related papers  (2025-01-23T08:33:10Z)
• StreamChat: Chatting with Streaming Video [85.02875830683637]
  StreamChat is a novel approach that enhances the interaction capabilities of Large Multimodal Models with streaming video content. We introduce a flexible and efficient crossattention-based architecture to process dynamic streaming inputs. We construct a new dense instruction dataset to facilitate the training of streaming interaction models.
  arXiv  Detail & Related papers  (2024-12-11T18:59:54Z)
• VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos [58.765796160750504]
  VideoGLaMM is a new model for fine-grained pixel-level grounding in videos based on user-provided textual inputs. The architecture is trained to synchronize both spatial and temporal elements of video content with textual instructions. Experimental results show that our model consistently outperforms existing approaches across all three tasks.
  arXiv  Detail & Related papers  (2024-11-07T17:59:27Z)
• Grounding Partially-Defined Events in Multimodal Data [61.0063273919745]
  We introduce a multimodal formulation for partially-defined events and cast the extraction of these events as a three-stage span retrieval task. We propose a benchmark for this task, MultiVENT-G, that consists of 14.5 hours of densely annotated current event videos and 1,168 text documents, containing 22.8K labeled event-centric entities. Results illustrate the challenges that abstract event understanding poses and demonstrates promise in event-centric video-language systems.
  arXiv  Detail & Related papers  (2024-10-07T17:59:48Z)
• The Surprising Effectiveness of Multimodal Large Language Models for Video Moment Retrieval [36.516226519328015]
  Video-language tasks necessitate spatial and temporal comprehension and require significant compute. This work demonstrates the surprising effectiveness of leveraging image-text pretrained MLLMs for moment retrieval. We achieve a new state-of-the-art in moment retrieval on the widely used benchmarks Charades-STA, QVHighlights, and ActivityNet Captions.
  arXiv  Detail & Related papers  (2024-06-26T06:59:09Z)
• Localizing Events in Videos with Multimodal Queries [61.20556229245365]
  Localizing events in videos based on semantic queries is a pivotal task in video understanding. We introduce ICQ, a new benchmark designed for localizing events in videos with multimodal queries. We propose 3 Multimodal Query Adaptation methods and a novel Surrogate Fine-tuning on pseudo-MQs strategy.
  arXiv  Detail & Related papers  (2024-06-14T14:35:58Z)
• Enabling Weakly-Supervised Temporal Action Localization from On-Device Learning of the Video Stream [5.215681853828831]
  We propose an efficient video learning approach to learn from a long, untrimmed streaming video. To the best of our knowledge, we are the first attempt to directly learn from the on-device, long video stream.
  arXiv  Detail & Related papers  (2022-08-25T13:41:03Z)
• VID-WIN: Fast Video Event Matching with Query-Aware Windowing at the Edge for the Internet of Multimedia Things [3.222802562733787]
  VID-WIN is an adaptive 2-stage allied windowing approach to accelerate video event analytics in an edge-cloud paradigm. VID-WIN exploits the video content and input knobs to accelerate the video inference process across nodes.
  arXiv  Detail & Related papers  (2021-04-27T10:08:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.