MVAFormer: RGB-based Multi-View Spatio-Temporal Action Recognition with Transformer

• URL: http://arxiv.org/abs/2511.02473v1
• Date: Tue, 04 Nov 2025 10:59:11 GMT
• Title: MVAFormer: RGB-based Multi-View Spatio-Temporal Action Recognition with Transformer
• Authors: Taiga Yamane, Satoshi Suzuki, Ryo Masumura, Shotaro Tora,
• Abstract summary: Multi-view action recognition aims to recognize human actions using multiple camera views.<n>Previous studies have explored promising cooperation methods for improving performance.<n>This paper proposes a multi-view action recognition method for the STAR setting, called MVAFormer.
• Score: 15.749459698197947
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-view action recognition aims to recognize human actions using multiple camera views and deals with occlusion caused by obstacles or crowds. In this task, cooperation among views, which generates a joint representation by combining multiple views, is vital. Previous studies have explored promising cooperation methods for improving performance. However, since their methods focus only on the task setting of recognizing a single action from an entire video, they are not applicable to the recently popular spatio-temporal action recognition~(STAR) setting, in which each person's action is recognized sequentially. To address this problem, this paper proposes a multi-view action recognition method for the STAR setting, called MVAFormer. In MVAFormer, we introduce a novel transformer-based cooperation module among views. In contrast to previous studies, which utilize embedding vectors with lost spatial information, our module utilizes the feature map for effective cooperation in the STAR setting, which preserves the spatial information. Furthermore, in our module, we divide the self-attention for the same and different views to model the relationship between multiple views effectively. The results of experiments using a newly collected dataset demonstrate that MVAFormer outperforms the comparison baselines by approximately $4.4$ points on the F-measure.

Related papers

• MAVR-Net: Robust Multi-View Learning for MAV Action Recognition with Cross-View Attention [0.5156484100374058]
  This paper presents MAVR-Net, a multi-view learning-based MAV action recognition framework.<n>Unlike traditional single-view methods, the proposed approach combines three complementary types of data, including raw frames, optical flow, and RGB segmentation masks.<n>Specifically, ResNet-based encoders are used to extract discnative features from each view, and a multi-scale feature pyramid is adopted to preserve the details of MAV motion patterns.
  arXiv  Detail & Related papers  (2025-10-17T09:04:51Z)
• JARViS: Detecting Actions in Video Using Unified Actor-Scene Context Relation Modeling [8.463489896549161]
  Two-stage Video localization (VAD) is a formidable task that involves the localization and classification of actions within the spatial and temporal dimensions of a video clip. We propose a two-stage VAD framework called Joint Actor-scene context Relation modeling (JARViS) JARViS consolidates cross-modal action semantics distributed globally across spatial and temporal dimensions using Transformer attention.
  arXiv  Detail & Related papers  (2024-08-07T08:08:08Z)
• Multi-view Action Recognition via Directed Gromov-Wasserstein Discrepancy [12.257725479880458]
  Action recognition has become one of the popular research topics in computer vision. We propose a multi-view attention consistency method that computes the similarity between two attentions from two different views of the action videos. Our approach applies the idea of Neural Radiance Field to implicitly render the features from novel views when training on single-view datasets.
  arXiv  Detail & Related papers  (2024-05-02T14:43:21Z)
• DVANet: Disentangling View and Action Features for Multi-View Action Recognition [56.283944756315066]
  We present a novel approach to multi-view action recognition where we guide learned action representations to be separated from view-relevant information in a video. Our model and method of training significantly outperforms all other uni-modal models on four multi-view action recognition datasets.
  arXiv  Detail & Related papers  (2023-12-10T01:19:48Z)
• Human-to-Human Interaction Detection [3.00604614803979]
  We introduce a new task named human-to-human interaction detection (HID) HID devotes to detecting subjects, recognizing person-wise actions, and grouping people according to their interactive relations, in one model. First, based on the popular AVA dataset created for action detection, we establish a new HID benchmark, termed AVA-Interaction (AVA-I)
  arXiv  Detail & Related papers  (2023-07-02T03:24:58Z)
• DOAD: Decoupled One Stage Action Detection Network [77.14883592642782]
  Localizing people and recognizing their actions from videos is a challenging task towards high-level video understanding. Existing methods are mostly two-stage based, with one stage for person bounding box generation and the other stage for action recognition. We present a decoupled one-stage network dubbed DOAD, to improve the efficiency for-temporal action detection.
  arXiv  Detail & Related papers  (2023-04-01T08:06:43Z)
• Revisiting spatio-temporal layouts for compositional action recognition [63.04778884595353]
  We take an object-centric approach to action recognition. The main focus of this paper is compositional/few-shot action recognition. We demonstrate how to improve the performance of appearance-based models by fusion with layout-based models.
  arXiv  Detail & Related papers  (2021-11-02T23:04:39Z)
• Motion Guided Attention Fusion to Recognize Interactions from Videos [40.1565059238891]
  We present a dual-pathway approach for recognizing fine-grained interactions from videos. We fuse the bottom-up features in the motion pathway with features captured from object detections to learn the temporal aspects of an action. We show that our approach can generalize across appearance effectively and recognize actions where an actor interacts with previously unseen objects.
  arXiv  Detail & Related papers  (2021-04-01T17:44:34Z)
• Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion [68.45737688496654]
  We present a modular interactive VOS framework which decouples interaction-to-mask and mask propagation. We show that our method outperforms current state-of-the-art algorithms while requiring fewer frame interactions.
  arXiv  Detail & Related papers  (2021-03-14T14:39:08Z)
• Pose And Joint-Aware Action Recognition [87.4780883700755]
  We present a new model for joint-based action recognition, which first extracts motion features from each joint separately through a shared motion encoder. Our joint selector module re-weights the joint information to select the most discriminative joints for the task. We show large improvements over the current state-of-the-art joint-based approaches on JHMDB, HMDB, Charades, AVA action recognition datasets.
  arXiv  Detail & Related papers  (2020-10-16T04:43:34Z)
• Collaborative Attention Mechanism for Multi-View Action Recognition [75.33062629093054]
  We propose a collaborative attention mechanism (CAM) for solving the multi-view action recognition problem. The proposed CAM detects the attention differences among multi-view, and adaptively integrates frame-level information to benefit each other. Experiments on four action datasets illustrate the proposed CAM achieves better results for each view and also boosts multi-view performance.
  arXiv  Detail & Related papers  (2020-09-14T17:33:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.