TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring

• URL: http://arxiv.org/abs/2508.04943v1
• Date: Thu, 07 Aug 2025 00:17:45 GMT
• Title: TRKT: Weakly Supervised Dynamic Scene Graph Generation with Temporal-enhanced Relation-aware Knowledge Transferring
• Authors: Zhu Xu, Ting Lei, Zhimin Li, Guan Wang, Qingchao Chen, Yuxin Peng, Yang liu,
• Abstract summary: Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships.<n>Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training.<n>We propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios.
• Score: 43.25899008682682
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dynamic Scene Graph Generation (DSGG) aims to create a scene graph for each video frame by detecting objects and predicting their relationships. Weakly Supervised DSGG (WS-DSGG) reduces annotation workload by using an unlocalized scene graph from a single frame per video for training. Existing WS-DSGG methods depend on an off-the-shelf external object detector to generate pseudo labels for subsequent DSGG training. However, detectors trained on static, object-centric images struggle in dynamic, relation-aware scenarios required for DSGG, leading to inaccurate localization and low-confidence proposals. To address the challenges posed by external object detectors in WS-DSGG, we propose a Temporal-enhanced Relation-aware Knowledge Transferring (TRKT) method, which leverages knowledge to enhance detection in relation-aware dynamic scenarios. TRKT is built on two key components:(1)Relation-aware knowledge mining: we first employ object and relation class decoders that generate category-specific attention maps to highlight both object regions and interactive areas. Then we propose an Inter-frame Attention Augmentation strategy that exploits optical flow for neighboring frames to enhance the attention maps, making them motion-aware and robust to motion blur. This step yields relation- and motion-aware knowledge mining for WS-DSGG. (2) we introduce a Dual-stream Fusion Module that integrates category-specific attention maps into external detections to refine object localization and boost confidence scores for object proposals. Extensive experiments demonstrate that TRKT achieves state-of-the-art performance on Action Genome dataset. Our code is avaliable at https://github.com/XZPKU/TRKT.git.

Related papers

• Spatio-temporal Graph Learning on Adaptive Mined Key Frames for High-performance Multi-Object Tracking [5.746443489229576]
  Key Frame Extraction (KFE) module leverages reinforcement learning to adaptively segment videos.<n> Intra-Frame Feature Fusion (IFF) module uses a Graph Convolutional Network (GCN) to facilitate information exchange between the target and surrounding objects.<n>Our proposed tracker achieves impressive results on the MOT17 dataset.
  arXiv  Detail & Related papers  (2025-01-17T11:36:38Z)
• Hierarchical Graph Interaction Transformer with Dynamic Token Clustering for Camouflaged Object Detection [57.883265488038134]
  We propose a hierarchical graph interaction network termed HGINet for camouflaged object detection. The network is capable of discovering imperceptible objects via effective graph interaction among the hierarchical tokenized features. Our experiments demonstrate the superior performance of HGINet compared to existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-08-27T12:53:25Z)
• EGTR: Extracting Graph from Transformer for Scene Graph Generation [5.935927309154952]
  Scene Graph Generation (SGG) is a challenging task of detecting objects and predicting relationships between objects. We propose a lightweight one-stage SGG model that extracts the relation graph from the various relationships learned in the multi-head self-attention layers of the DETR decoder. We demonstrate the effectiveness and efficiency of our method for the Visual Genome and Open Image V6 datasets.
  arXiv  Detail & Related papers  (2024-04-02T16:20:02Z)
• Building Category Graphs Representation with Spatial and Temporal Attention for Visual Navigation [35.13932194789583]
  Given an object of interest, visual navigation aims to reach the object's location based on a sequence of partial observations. To this end, an agent needs to 1) learn a piece of certain knowledge about the relations of object categories in the world during training and 2) look for the target object based on the pre-learned object category relations and its moving trajectory in the current unseen environment. We propose a Category Relation Graph (CRG) to learn the knowledge of object category layout relations and a Temporal-Spatial-Region (TSR) attention architecture to perceive the long-term spatial-temporal dependencies of objects helping the
  arXiv  Detail & Related papers  (2023-12-06T07:28:43Z)
• Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuning [84.39787427288525]
  Scene graph generation (SGG) is a fundamental task aimed at detecting visual relations between objects in an image. We introduce open-vocabulary scene graph generation, a novel, realistic and challenging setting in which a model is trained on a set of base object classes. Our method can support inference over completely unseen object classes, which existing methods are incapable of handling.
  arXiv  Detail & Related papers  (2022-08-17T09:05:38Z)
• Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generation [33.15624351965304]
  We argue that most existing WSSGG works only focus on object-consistency. We propose to enhance a simple grounding module with both object-aware and interaction-aware knowledge. Our method consistently improves WSSGG performance on various kinds of weak supervision.
  arXiv  Detail & Related papers  (2022-08-03T04:20:17Z)
• Relation Regularized Scene Graph Generation [206.76762860019065]
  Scene graph generation (SGG) is built on top of detected objects to predict object pairwise visual relations. We propose a relation regularized network (R2-Net) which can predict whether there is a relationship between two objects. Our R2-Net can effectively refine object labels and generate scene graphs.
  arXiv  Detail & Related papers  (2022-02-22T11:36:49Z)
• Exploiting Scene Graphs for Human-Object Interaction Detection [81.49184987430333]
  Human-Object Interaction (HOI) detection is a fundamental visual task aiming at localizing and recognizing interactions between humans and objects. We propose a novel method to exploit this information, through the scene graph, for the Human-Object Interaction (SG2HOI) detection task. Our method, SG2HOI, incorporates the SG information in two ways: (1) we embed a scene graph into a global context clue, serving as the scene-specific environmental context; and (2) we build a relation-aware message-passing module to gather relationships from objects' neighborhood and transfer them into interactions.
  arXiv  Detail & Related papers  (2021-08-19T09:40:50Z)
• A Graph-based Interactive Reasoning for Human-Object Interaction Detection [71.50535113279551]
  We present a novel graph-based interactive reasoning model called Interactive Graph (abbr. in-Graph) to infer HOIs. We construct a new framework to assemble in-Graph models for detecting HOIs, namely in-GraphNet. Our framework is end-to-end trainable and free from costly annotations like human pose.
  arXiv  Detail & Related papers  (2020-07-14T09:29:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.