Transformed ROIs for Capturing Visual Transformations in Videos

• URL: http://arxiv.org/abs/2106.03162v1
• Date: Sun, 6 Jun 2021 15:59:53 GMT
• Title: Transformed ROIs for Capturing Visual Transformations in Videos
• Authors: Abhinav Rai, Fadime Sener, Angela Yao
• Abstract summary: We present TROI, a plug-and-play module for CNNs to reason between mid-level feature representations that are otherwise separated in space and time. We achieve state-of-the-art action recognition results on the large-scale datasets Something-Something-V2 and Epic-Kitchens-100.
• Score: 31.88528313257094
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modeling the visual changes that an action brings to a scene is critical for video understanding. Currently, CNNs process one local neighbourhood at a time, so contextual relationships over longer ranges, while still learnable, are indirect. We present TROI, a plug-and-play module for CNNs to reason between mid-level feature representations that are otherwise separated in space and time. The module relates localized visual entities such as hands and interacting objects and transforms their corresponding regions of interest directly in the feature maps of convolutional layers. With TROI, we achieve state-of-the-art action recognition results on the large-scale datasets Something-Something-V2 and Epic-Kitchens-100.

Related papers

• VrdONE: One-stage Video Visual Relation Detection [30.983521962897477]
  Video Visual Relation Detection (VidVRD) focuses on understanding how entities over time and space in videos. Traditional methods for VidVRD, challenged by its complexity, typically split the task into two parts: one for identifying what relation are present and another for determining their temporal boundaries. We propose VrdONE, a streamlined yet efficacious one-stage model for VidVRD.
  arXiv  Detail & Related papers  (2024-08-18T08:38:20Z)
• Structured Video-Language Modeling with Temporal Grouping and Spatial Grounding [112.3913646778859]
  We propose a simple yet effective video-language modeling framework, S-ViLM. It includes two novel designs, inter-clip spatial grounding and intra-clip temporal grouping, to promote learning region-object alignment and temporal-aware features. S-ViLM surpasses the state-of-the-art methods substantially on four representative downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T22:45:07Z)
• Continuous Scene Representations for Embodied AI [33.00565252990522]
  Continuous Scene Representations (CSR) is a scene representation constructed by an embodied agent navigating within a space. Our key insight is to embed pair-wise relationships between objects in a latent space. CSR can track objects as the agent moves in a scene, update the representation accordingly, and detect changes in room configurations.
  arXiv  Detail & Related papers  (2022-03-31T17:55:33Z)
• Contextual Attention Network: Transformer Meets U-Net [0.0]
  convolutional neural networks (CNN) have become the de facto standard and attained immense success in medical image segmentation. However, CNN based methods fail to build long-range dependencies and global context connections. Recent articles have exploited Transformer variants for medical image segmentation tasks.
  arXiv  Detail & Related papers  (2022-03-02T21:10:24Z)
• Video-Text Pre-training with Learned Regions [59.30893505895156]
  Video-Text pre-training aims at learning transferable representations from large-scale video-text pairs. We propose a module for videotext-learning, RegionLearner, which can take into account the structure of objects during pre-training on large-scale video-text pairs.
  arXiv  Detail & Related papers  (2021-12-02T13:06:53Z)
• Object-Region Video Transformers [100.23380634952083]
  We present Object-Region Transformers Video (ORViT), an emphobject-centric approach that extends transformer video layers with object representations. Our ORViT block consists of two object-level streams: appearance and dynamics. We show strong improvement in performance across all tasks and considered, demonstrating the value of a model that incorporates object representations into a transformer architecture.
  arXiv  Detail & Related papers  (2021-10-13T17:51:46Z)
• EAN: Event Adaptive Network for Enhanced Action Recognition [66.81780707955852]
  We propose a unified action recognition framework to investigate the dynamic nature of video content. First, when extracting local cues, we generate the spatial-temporal kernels of dynamic-scale to adaptively fit the diverse events. Second, to accurately aggregate these cues into a global video representation, we propose to mine the interactions only among a few selected foreground objects by a Transformer.
  arXiv  Detail & Related papers  (2021-07-22T15:57:18Z)
• Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers [77.52828273633646]
  We present a new drop-in block for video transformers that aggregates information along implicitly determined motion paths. We also propose a new method to address the quadratic dependence of computation and memory on the input size. We obtain state-of-the-art results on the Kinetics, Something--Something V2, and Epic-Kitchens datasets.
  arXiv  Detail & Related papers  (2021-06-09T21:16:05Z)
• IAUnet: Global Context-Aware Feature Learning for Person Re-Identification [106.50534744965955]
  IAU block enables the feature to incorporate the globally spatial, temporal, and channel context. It is lightweight, end-to-end trainable, and can be easily plugged into existing CNNs to form IAUnet. Experiments show that IAUnet performs favorably against state-of-the-art on both image and video reID tasks.
  arXiv  Detail & Related papers  (2020-09-02T13:07:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.