ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition

• URL: http://arxiv.org/abs/2410.05774v4
• Date: Mon, 11 Nov 2024 17:06:25 GMT
• Title: ActionAtlas: A VideoQA Benchmark for Domain-specialized Action Recognition
• Authors: Mohammadreza Salehi, Jae Sung Park, Tanush Yadav, Aditya Kusupati, Ranjay Krishna, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi,
• Abstract summary: ActionAtlas is a video question answering benchmark featuring short videos across various sports. The dataset includes 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices. We evaluate open and proprietary foundation models on this benchmark, finding that the best model, GPT-4o, achieves a maximum accuracy of 45.52%.
• Score: 111.32822459456793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Our world is full of varied actions and moves across specialized domains that we, as humans, strive to identify and understand. Within any single domain, actions can often appear quite similar, making it challenging for deep models to distinguish them accurately. To evaluate the effectiveness of multimodal foundation models in helping us recognize such actions, we present ActionAtlas v1.0, a multiple-choice video question answering benchmark featuring short videos across various sports. Each video in the dataset is paired with a question and four or five choices. The question pinpoints specific individuals, asking which choice "best" describes their action within a certain temporal context. Overall, the dataset includes 934 videos showcasing 580 unique actions across 56 sports, with a total of 1896 actions within choices. Unlike most existing video question answering benchmarks that only cover simplistic actions, often identifiable from a single frame, ActionAtlas focuses on intricate movements and rigorously tests the model's capability to discern subtle differences between moves that look similar within each domain. We evaluate open and proprietary foundation models on this benchmark, finding that the best model, GPT-4o, achieves a maximum accuracy of 45.52%. Meanwhile, Non-expert crowd workers, provided with action description for each choice, achieve 61.64% accuracy, where random chance is approximately 21%. Our findings with state-of-the-art models indicate that having a high frame sampling rate is important for accurately recognizing actions in ActionAtlas, a feature that some leading proprietary video models, such as Gemini, do not include in their default configuration.

Related papers

• Video Action Differencing [92.3218782696305]
  We introduce Video Action Differencing (VidDiff), the novel task of identifying subtle differences between videos of the same action. We first create VidDiffBench, a benchmark dataset containing 549 video pairs. Our experiments demonstrate that VidDiffBench poses a significant challenge for state-of-the-art large multimodal models.
  arXiv  Detail & Related papers  (2025-03-10T21:18:32Z)
• Early Action Recognition with Action Prototypes [62.826125870298306]
  We propose a novel model that learns a prototypical representation of the full action for each class. We decompose the video into short clips, where a visual encoder extracts features from each clip independently. Later, a decoder aggregates together in an online fashion features from all the clips for the final class prediction.
  arXiv  Detail & Related papers  (2023-12-11T18:31:13Z)
• Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features [0.0]
  Group-level emotion recognition can be useful in many fields including social robotics, conversational agents, e-coaching and learning analytics. This paper explores privacy-compliant group-level emotion recognition ''in-the-wild'' within the EmotiW Challenge 2023.
  arXiv  Detail & Related papers  (2023-12-06T08:58:11Z)
• Video-Specific Query-Key Attention Modeling for Weakly-Supervised Temporal Action Localization [14.43055117008746]
  Weakly-trimmed temporal action localization aims to identify and localize the action instances in the unsupervised videos with only video-level action labels. We propose a network named VQK-Net with a video-specific query-key attention modeling that learns a unique query for each action category of each input video.
  arXiv  Detail & Related papers  (2023-05-07T04:18:22Z)
• A Graph-Based Method for Soccer Action Spotting Using Unsupervised Player Classification [75.93186954061943]
  Action spotting involves understanding the dynamics of the game, the complexity of events, and the variation of video sequences. In this work, we focus on the former by (a) identifying and representing the players, referees, and goalkeepers as nodes in a graph, and by (b) modeling their temporal interactions as sequences of graphs. For the player identification task, our method obtains an overall performance of 57.83% average-mAP by combining it with other modalities.
  arXiv  Detail & Related papers  (2022-11-22T15:23:53Z)
• ASOD60K: Audio-Induced Salient Object Detection in Panoramic Videos [79.05486554647918]
  We propose PV-SOD, a new task that aims to segment salient objects from panoramic videos. In contrast to existing fixation-level or object-level saliency detection tasks, we focus on multi-modal salient object detection (SOD) We collect the first large-scale dataset, named ASOD60K, which contains 4K-resolution video frames annotated with a six-level hierarchy.
  arXiv  Detail & Related papers  (2021-07-24T15:14:20Z)
• MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actions [39.27858380391081]
  This paper aims to present a new multi-person dataset of atomic-temporal actions, coined as MultiSports. We build the dataset of MultiSports v1.0 by selecting 4 sports classes, collecting around 3200 video clips, and annotating around 37790 action instances with 907k bounding boxes.
  arXiv  Detail & Related papers  (2021-05-16T10:40:30Z)
• Hybrid Dynamic-static Context-aware Attention Network for Action Assessment in Long Videos [96.45804577283563]
  We present a novel hybrid dynAmic-static Context-aware attenTION NETwork (ACTION-NET) for action assessment in long videos. We learn the video dynamic information but also focus on the static postures of the detected athletes in specific frames. We combine the features of the two streams to regress the final video score, supervised by ground-truth scores given by experts.
  arXiv  Detail & Related papers  (2020-08-13T15:51:42Z)
• Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA [96.10612095576333]
  We propose a video question answering model which effectively integrates multi-modal input sources and finds the temporally relevant information to answer questions. Our model is also comprised of dual-level attention (word/object and frame level), multi-head self-cross-integration for different sources (video and dense captions), and which pass more relevant information to gates. We evaluate our model on the challenging TVQA dataset, where each of our model components provides significant gains, and our overall model outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2020-05-13T16:35:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.