VrdONE: One-stage Video Visual Relation Detection

• URL: http://arxiv.org/abs/2408.09408v2
• Date: Wed, 16 Oct 2024 11:28:19 GMT
• Title: VrdONE: One-stage Video Visual Relation Detection
• Authors: Xinjie Jiang, Chenxi Zheng, Xuemiao Xu, Bangzhen Liu, Weiying Zheng, Huaidong Zhang, Shengfeng He,
• Abstract summary: Video Visual Relation Detection (VidVRD) focuses on understanding how entities over time and space in videos. Traditional methods for VidVRD, challenged by its complexity, typically split the task into two parts: one for identifying what relation are present and another for determining their temporal boundaries. We propose VrdONE, a streamlined yet efficacious one-stage model for VidVRD.
• Score: 30.983521962897477
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video Visual Relation Detection (VidVRD) focuses on understanding how entities interact over time and space in videos, a key step for gaining deeper insights into video scenes beyond basic visual tasks. Traditional methods for VidVRD, challenged by its complexity, typically split the task into two parts: one for identifying what relation categories are present and another for determining their temporal boundaries. This split overlooks the inherent connection between these elements. Addressing the need to recognize entity pairs' spatiotemporal interactions across a range of durations, we propose VrdONE, a streamlined yet efficacious one-stage model. VrdONE combines the features of subjects and objects, turning predicate detection into 1D instance segmentation on their combined representations. This setup allows for both relation category identification and binary mask generation in one go, eliminating the need for extra steps like proposal generation or post-processing. VrdONE facilitates the interaction of features across various frames, adeptly capturing both short-lived and enduring relations. Additionally, we introduce the Subject-Object Synergy (SOS) module, enhancing how subjects and objects perceive each other before combining. VrdONE achieves state-of-the-art performances on the VidOR benchmark and ImageNet-VidVRD, showcasing its superior capability in discerning relations across different temporal scales. The code is available at https://github.com/lucaspk512/vrdone.

Related papers

• HyRSM++: Hybrid Relation Guided Temporal Set Matching for Few-shot Action Recognition [51.2715005161475]
  We propose a novel Hybrid Relation guided temporal Set Matching approach for few-shot action recognition. The core idea of HyRSM++ is to integrate all videos within the task to learn discriminative representations. We show that our method achieves state-of-the-art performance under various few-shot settings.
  arXiv  Detail & Related papers  (2023-01-09T13:32:50Z)
• Tag-Based Attention Guided Bottom-Up Approach for Video Instance Segmentation [83.13610762450703]
  Video instance is a fundamental computer vision task that deals with segmenting and tracking object instances across a video sequence. We introduce a simple end-to-end train bottomable-up approach to achieve instance mask predictions at the pixel-level granularity, instead of the typical region-proposals-based approach. Our method provides competitive results on YouTube-VIS and DAVIS-19 datasets, and has minimum run-time compared to other contemporary state-of-the-art performance methods.
  arXiv  Detail & Related papers  (2022-04-22T15:32:46Z)
• Exploring Motion and Appearance Information for Temporal Sentence Grounding [52.01687915910648]
  We propose a Motion-Appearance Reasoning Network (MARN) to solve temporal sentence grounding. We develop separate motion and appearance branches to learn motion-guided and appearance-guided object relations. Our proposed MARN significantly outperforms previous state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2022-01-03T02:44:18Z)
• Learning Spatial-Temporal Graphs for Active Speaker Detection [26.45877018368872]
  SPELL is a framework that learns long-range multimodal graphs to encode the inter-modal relationship between audio and visual data. We first construct a graph from a video so that each node corresponds to one person. We demonstrate that learning graph-based representation, owing to its explicit spatial and temporal structure, significantly improves the overall performance.
  arXiv  Detail & Related papers  (2021-12-02T18:29:07Z)
• Spatio-Temporal Interaction Graph Parsing Networks for Human-Object Interaction Recognition [55.7731053128204]
  In given video-based Human-Object Interaction scene, modeling thetemporal relationship between humans and objects are the important cue to understand the contextual information presented in the video. With the effective-temporal relationship modeling, it is possible not only to uncover contextual information in each frame but also directly capture inter-time dependencies. The full use of appearance features, spatial location and the semantic information are also the key to improve the video-based Human-Object Interaction recognition performance.
  arXiv  Detail & Related papers  (2021-08-19T11:57:27Z)
• Learning Visual Affordance Grounding from Demonstration Videos [76.46484684007706]
  Affordance grounding aims to segment all possible interaction regions between people and objects from an image/video. We propose a Hand-aided Affordance Grounding Network (HAGNet) that leverages the aided clues provided by the position and action of the hand in demonstration videos.
  arXiv  Detail & Related papers  (2021-08-12T11:45:38Z)
• What and When to Look?: Temporal Span Proposal Network for Video Visual Relation Detection [4.726777092009554]
  Video Visual Relation Detection (VidD): segment-based and window-based. We first point out the limitations these two methods have and propose Temporal Span Proposal Network (TSPN), a novel method with two advantages in terms of efficiency and effectiveness.
  arXiv  Detail & Related papers  (2021-07-15T07:01:26Z)
• Relation-aware Hierarchical Attention Framework for Video Question Answering [6.312182279855817]
  We propose a novel Relation-aware Hierarchical Attention (RHA) framework to learn both the static and dynamic relations of the objects in videos. In particular, videos and questions are embedded by pre-trained models firstly to obtain the visual and textual features. We consider the temporal, spatial, and semantic relations, and fuse the multimodal features by hierarchical attention mechanism to predict the answer.
  arXiv  Detail & Related papers  (2021-05-13T09:35:42Z)
• LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos [13.25502885135043]
  Analyzing the interactions between humans and objects from a video includes identification of relationships between humans and the objects present in the video. We present a hierarchical approach, LIGHTEN, to learn visual features to effectively capture truth at multiple granularities in a video. We achieve state-of-the-art results in human-object interaction detection (88.9% and 92.6%) and anticipation tasks of CAD-120 and competitive results on image based HOI detection in V-COCO.
  arXiv  Detail & Related papers  (2020-12-17T05:44:07Z)
• Co-Saliency Spatio-Temporal Interaction Network for Person Re-Identification in Videos [85.6430597108455]
  We propose a novel Co-Saliency Spatio-Temporal Interaction Network (CSTNet) for person re-identification in videos. It captures the common salient foreground regions among video frames and explores the spatial-temporal long-range context interdependency from such regions. Multiple spatialtemporal interaction modules within CSTNet are proposed, which exploit the spatial and temporal long-range context interdependencies on such features and spatial-temporal information correlation.
  arXiv  Detail & Related papers  (2020-04-10T10:23:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.